Image for post
Image for post

Chunking your lists into multiple smaller lists

As part of the development of our new app feature, we are adding the ability to allow users to track their journeys. They can Start / Stop the journey tracking and allow the app to record their distance, time taken etc. This is primarily to be used to allow users to support mileage claims.

A journey takes the form of a Model containing properties for storing the user, start date, end date, mileage etc. The journey also contains a list of waypoints. These are the longitude / latitude points that are generated by the user’s position. A waypoint is taken every 5 seconds and added to the list of waypoints for the journey. From these waypoints we can then generate a map of their journey and display this to them.

During initial testing everything was fine as we tested the feature on smaller journeys containing a few hundred waypoints. As we began stress testing the feature, we noticed we were getting timouts as we started to exceed 800 or so waypoints. By 1000 waypoints were getting regular timeouts. We discovered the reason was due to the volume of waypoints we were posting back to our service. As the number of waypoints grew, the time taken to POST the data over our RESTful service grew too. And this was causing our timeout problem.

I investigated several options, but the cleanest and most simple was to chunk the waypoints into smaller discrete lists which we would POST. So instead of POSTing all of the waypoints in one large payload, we would instead send multiple smaller payloads.

So how do you chunk your list into a list of smaller lists? There are many ways of achieving this, and I’m sure those of you reading this article will be able to suggest your own versions of the algorithm I have used here. Firstly, instead of using a hard-coded version that only works with journey waypoint lists, I have implemented an extension method that can work with any type of list. This allows me to chunk any type of list data going forwards (I have already got a few ideas in mind of how I will reuse this extension method).

Hide Copy Code

So we can see that what is returned is a list of lists of type T. The extension method is applied to the source list (which is to be chunked into smaller lists). The parameter to the extension method is the number of items to appear in each chunked list. The implementation uses the LINQ methods of Skip() and Take() to iterate over the list. The Skip() method will ignore the first n items in the list. The Take() method will then take the next n elements from the list. These methods therefore when used in conjunction easily iterate over our list. The use of yield return helps to iterate over the list in an efficient manner by processing the next item without having to process the entire list (lazy evaluation).

In our specific case for chunking our journey waypoints, we have set the chunking value to 500. Although the problem didn’t appear until at least 800 items, I wanted to keep the value to a safe, low limit just to err on the side of caution.

Here is the code from one of the unit tests I’ve written that exercises the chunking extension method.

Hide Expand

Image for post
Image for post

Copy Code

So in summary, if you are dealing with large lists of items and need to break them down in smaller, more manageable lists, then chunking them is a simple and very effective solution. This works great in our mobile app where we are sending large lists of data to a backend service using the resource hungry environment of the smart phone where memory and processing power are in short supply. Processing numerous smaller lists is more efficient (and less error prone) than trying to process one large list. It also uses less resources (memory, CPU) to do so.

A father, cyclist, vegetarian, atheist, geek and multiple award winning technical author. Loves real ale, fine wine and good music. All round decent chap.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store