Handling deep paging efficiently

In most cases, the top results returned to the user should be what they are looking for. The top results should be the most relevant ones and the ones we want to show. However, there are use cases where this is not enough. Sometimes we want to get all the results—in the worst case, we want to get all the documents stored in the collection and do something with them. When you are requesting a high number of pages, you will see that the performance will start suffering. This is because Solr needs to build the results list for each request and discard the first N ones to get to the requested page. Of course, there are better ways to handle such cases, and Solr allows you to use one of those methods that we will discuss in this recipe.

How to do it...

1. The actual index structure doesn't matter, but for the purpose of this recipe, let's assume that we have the following index structure (we just put the field's definition in the schema.xml file):

   <field name="id...