Understanding soft commit, optimize, and hard commit

Solr provides us a Near-Real-Time (NRT) search, which makes documents available for searching just after they have been indexed in Solr. Additions or updates to documents are seen nearly in real-time after we index them in Solr. This near-real-time search can be done by using a soft commit (available in Solr 4.0+), which avoids the high cost of calling fsync, and it will flush the index data into a stable storage so that it can be retrieved in the event of a JVM crash.

An optimize, on the other hand, will cause all index segments to be merged into a single segment first and then reindex them. It's just like the defragmentation that we do on an HDD, which reindexes and frees up space. Normally, index segments are merged over time as specified in the merge policy, but this happens immediately when forced using the optimize command.

Let's see how we can use soft commit and optimize in Solr. We'll use our musicCatalog example and create a new...