Understanding the limits

Although you can have a completely distributed system for your big data search, there is a limit in terms of how far you can go. As you keep on distributing the shards, you may end up facing what is called the "laggard problem" for indexes for your instance.

This problem states that the response to your search query, which is an aggregation of results from all the shards, is controlled by the following formula:

QueryResponse = avg(max(shardResponseTime))

This means that if you have many shards, it is more likely that you will have one of them responding slowly (due to some anomaly) to your queries, and this will impact on your query response time, and this will start increasing.

The distributed search in Apache Solr has many limitations. Each document uploaded as distributed big data must have a unique key, and this unique key must be stored in the Solr repository. To do so, the Solr schema.xml file should have "stored=true" against the key attribute. This unique key...