Administrating Solr: Master the use of Drupal and associated scripts to administrate, monitor, and optimize Solr

Before we get ready for the installation, you need to have the necessary downloads ready.

Once you have the mentioned installers ready, you may proceed installing them as follows:

Let us understand what a request and response stands for and get a brief idea about the components handling these requests.

Requests are handled using multiple handlers and/or multiple instances of the same SolrRequestHandler class. How do you wish to use the handler and instances of the handler class is differentiated based on the custom configurations, and are registered with SolrCore. An alternate way to register your SolrRequestHandler with the core is through the solrconfig.xml file.

For instance:

<requestHandler name="/foo" class="solr.CustomRequestHandler" />
    <!-- initialization args may optionally be defined here -->
     <lst name="defaults">
       <int name="rows">10</int>
       <str name="fl">*</str>
       <str name="version">2.1</str>
     </lst>
  </requestHandler>

The easiest way to implement SolrRequestHandler is to extend the RequestHandlerBase class.

Writing a simple query is definitely an easy job; however, writing a complex one with queries playing around with phrases, boosting and prioritizing search results, nesting your query, and a search even based on partial match would be a challenging task. In addition to this, you must remember to write your query taking the performance aspects into account. This is one of the reasons why something that seems to be simple at first sight, actually proves to be even more challenging like writing a complex query which is equally good and efficient in terms of performance. This chapter will guide you through a few of the tasks you are expected to encounter during your everyday work with Solr.

<field name="id" type="string" indexed="true" stored="true" required="true" /> 
<field name="title" type="text" indexed="true" stored="true" /> 
<field name="author" type="string" indexed="true" stored="true"/>

There might be situations wherein you need to search a document title within millions of documents for which string based search is of course not a good idea. So, the question for ourselves; is it possible to achieve using Solr? Fortunately, yes and the next example will guide you through it.

Assume that you have the following type defined, that needs to be added to your schema.xml file.

<fieldType name="text" class="solr.TextField" positionIncrementGap="100"> 
<analyzer> 
<tokenizer class="solr.WhitespaceTokenizerFactory"/> 
<filter class="solr.LowerCaseFilterFactory"/> 
<filter class="solr.SnowballPorterFilterFactory" language="English"/> 
</analyzer> 
</fieldType>

And then, add the following fields to your schema.xml.

<field name="id" type="string" indexed="true" stored="true" required="true" /> 
<field name="title" type="text" indexed="true" stored="true" />

Assume that your data looks like this:

<add> 
<doc>
<field name="id">1</field> 
<field name="title">2012 report</field> 
</doc> 
<doc> 
<field name="id">2</field> 
<field name="title">2007 report</field> 
</doc> 
<doc> 
<field name="id">3</field> 
<field name="title">2012 draft report</field> 
</doc> 
</add>

Now, let us instruct Solr to find the documents that have a 2012 report phrase embedded in the title. Execute the following query to Solr:

http://localhost:8080/solr/select?q=title:"2012 report"

If you get the following result, bingo !!! your query worked!

<?xml version="1.0" encoding="UTF-8"?> 
<response> 
<lst name="responseHeader"> 
<int name="status">0</int> 
<int name="QTime">1</int> 
<lst name="params"> 
<str name="q">title:"2012 report"</str> 
</lst> 
</lst> 
<result name="response" numFound="1" start="0"> 
<doc> 
<str name="id">1</str> 
<str name="title">2012 report</str> 
</doc> 
</result> 
</response>

The debug query (the debugQuery=on parameter) shows us what lucene query was made:

<str name="parsedquery">PhraseQuery(title:"2012 report")</str>

As you must have noticed, we got just one document as a result of our query, omitting even the document with the title: 2012 draft report (which is very appropriate and perfect output).

We have used only two fields to demonstrate the concept due to the fact that we are more committed to search a phrase within the title field, here in this demonstration.

Interestingly, here standard Solr query parser has been queried; hence, the field name and the associated value we are looking for can be specified. The query differs from the standard word-search query by using the " character both at the start and end of the query. It dictates Solr to consider the search as a phrase query instead of a term query (which actually makes the difference!). So, this phrase query tells Solr to search considering all the words as a single unit, and not individually.

In addition to this, the phrase query just ensured that the phrase query (that is, the desired one) was made instead of the standard term query.

Since you are in a competitive market, assume that one day your online product met a disaster wherein your product's search result suddenly falls down. To overcome this scenario and survive in such a competitive market, probably you would like to favor documents that have the exact phrase typed by the end-user over the documents that have matches in separate words. We will guide you on how to achieve this in this section.

I assume that we will use dismax query parser, instead of the standard one. Moreover, we will re-use the same schema.xml that was demonstrated in the Searching for a phrase section in this chapter.

Our sample data looks like this:

<add> 
<doc> 
<field name="id">1</field> 
<field name="title">Annual 2012 report final draft</field> 
</doc> 
<doc> 
<field name="id">2</field> 
<field name="title">2007 report</field> 
</doc> 
<doc> 
<field name="id">3</field> 
<field name="title">2012 draft report</field> 
</doc> 
</add>

As mentioned earlier, we would like to boost or give preference to those documents that have phrase matches over others matching the query. To achieve this, run the following query to your Solr instance:

http://localhost:8080/solr/select?defType=dismax&pf=title^100&q=2012 +report&qf=title

And the desired result should look like:

<?xml version="1.0" encoding="UTF-8"?> 
<response> 
<lst name="responseHeader"> 
<int name="status">0</int> 
<int name="QTime">1</int> 
<lst name="params"> 
<str name="qf">title</str> 
<str name="pf">title^100</str> 
<str name="q">2012 report</str> 
<str name="defType">dismax</str>
</lst> 
</lst> 
<result name="response" numFound="2" start="0"> 
<doc> 
<str name="id">1</str> 
<str name="title">Annual 2012 report last draft</str> 
</doc> 
<doc> 
<str name="id">3</str> 
<str name="title">2012 draft report</str> 
</doc> 
</result> 
</response>

We have a couple of parameters which have been added to this example and might be new to you. Don't worry! I will explain all of them. The first parameter is defType, which tells Solr which query parser we will be using (dismax in our case). If you are not familiar or would like to learn more about dismax, http://wiki.apache.org/solr/DisMax is where you should go! One of the features of this query parser is the ability to tell Solr which field should be used to search for phrases, and this is achieved using the pf parameter. The pf parameter takes a list of fields with the boost that corresponds to them, for instance, pf=title^100 which means that the phrase found in the title field will be boosted with a value of 100. The q parameter is the standard query parameter which you might be familiar with. In our example, we passed the words we are searching for using AND operator. Through our example we are looking for the documents which satisfy '2012' AND 'report' equation, also known as occurrences of both '2012' and 'report' words found in the title.

You might come across situations wherein you need to promote some of your products and would like to find those on top of other documents in the search result list. Additionally, you might also need to have such products flexible and define exclusive queries applicable only to these products and not to the others. To achieve so, you might think of options such as boosting, index time boosting, or probably some special field. Don't worry! Solr will help you out via this section using a robust component known as QueryElevationComponent.

As QueryElevationComponent is biased to specific documents, it impacts the overall search process for other documents. Thus, it is recommended to use this feature only when it is required.

First of all, let us add the component definition in the solrconfig.xml file, which should look like this:

<searchComponent name="elevator" class="solr.QueryElevationComponent" > 
<str name="queryFieldType">string</str> 
<str name="config-file">elevate.xml</str> 
</searchComponent>

Now we will add the appropriate request handler that will include the elevation component. We will name it /promote it, due to the fact that this feature is mainly used to promote your document in search results. Add this to your solrconfig.xml file:

<requestHandler name="/promotion" class="solr.SearchHandler"> 
<arr name="last-components"> 
<str>elevator</str> 
</arr> 
</requestHandler>

You must have noticed a mysterious file, elevate.xml that has been included in the query elevation component, which actually contains the following data and are placed in the configuration directory of the Solr instance.

<?xml version="1.0" encoding="UTF-8" ?> 
<elevate> 
<query text="solr"> 
<doc id="3" /> 
<doc id="1" /> 
</query> 
</elevate>

Here we want our documents with identifiers 3 and 1 to be on the first and second position respectively in the search result list.

Now it is time to add the below field definition to the schema.xml file.

<field name="id" type="string" indexed="true" stored="true" required="true" /> 
<field name="name" type="text" indexed="true" stored="true" />

The following are the data which have been indexed:

<add> 
<doc> 
  <field name="id">1</field> 
  <field name="name">Solr Optimization</field> 
</doc> 
<doc>
  <field name="id">2</field> 
  <field name="name">Solr Monitoring</field> 
</doc> 
<doc> 
   <field name="id">3</field> 
   <field name="name">Solr annual report</field> 
</doc> 
</add>

Now, it's time to run the following query:

http://localhost:8080/solr/promotion?q=solr

If you get the following result, you can be assured that your query worked out successfully:

<?xml version="1.0" encoding="UTF-8"?> 
<response> 
<lst name="responseHeader"> 
<int name="status">0</int> 
<int name="QTime">1</int> 
<lst name="params"> 
<str name="q">solr</str> 
</lst> 
</lst> 
<result name="response" numFound="3" start="0"> 
<doc> 
<str name="id">3</str> 
<str name="name">Solr annual report</str> 
</doc> 
<doc> 
<str name="id">1</str> 
<str name="name">Solr Optimization</str> 
</doc> 
<doc> 
<str name="id">2</str> 
<str name="name">Solr Monitoring</str> 
</doc> 
</result> 
</response>

In the first part of the configuration, we have defined a new search component (elevator component in our case) and a class attribute (the QueryElevationComponent class in our case). Along with these, we have two additional attributes that define the elevation component behavior which are as follows:

Now, let us step into the next part of solrconfig.xml, which is search handler definition. It tells Solr to create a new search handler with the name /promotion (the name attribute) and using the solr.SearchHandler class (the class attribute). This handler definition also tells Solr to include a component named elevator, which means that the search handler is going to use our defined component. As you might know, you can use more than one search component in a single search handler.

In the actual configuration of the elevate component, you can see that there is a query defined (the query XML tag) with an attribute text="solr", which defines the behavior of the component when a user passes solr to the q parameter. You can see a list of unique identifiers of documents that will be placed on top of the results list for the defined query under this tag, where each document is defined by a doc tag and an id attribute (which have to be defined on the basis of solr.StrField) which holds the unique identifier.

The query is made to our new handler with just a simple one word q parameter (the default search field is set to name in the schema.xml file). Recall the elevate.xml file and the documents we defined for the query we just passed to Solr. Yes of course, we told Solr that we want documents with id=3 and id=1 to be placed on first and second positions respectively in the search result list. And ultimately, our query worked and you can see the documents were placed exactly as we wanted.

You might come across situations wherein you need to nest a query within another query. Let us imagine that you want to run a query using the standard request handler, but you need to embed a query that is parsed by the dismax query parser inside it. Isn't that interesting? We will show you how to do it.

Let us assume that we use the same field definition in schema.xml that was used in our previous section "Based on a partial keyword/phrase match".

Our example data looks like this:

<add> 
<doc> 
<field name="id">1</field> 
<field name="title">Reviewed solrcook book</field> 
</doc> 
<doc> 
<field name="id">2</field> 
<field name="title">Some book reviewed</field> 
</doc> 
<doc> 
<field name="id">3</field> 
<field name="title">Another reviewed little book</field> 
</doc> 
</add>

Here, we are going to use the standard query parser to support lucene query syntax, but we would like to boost phrases using the dismax query parser. At first it seems to be impossible to achieve, but don't worry, we will handle it. Let us suppose that we want to find books having the words, reviewed and book, in their title field; and we would like to boost the reviewed book phrase by 10. Here we go with the query:

http://localhost:8080/solr/select?q=reviewed+AND+book+AND+_ query_:"{!dismax qf=title pf=title^10 v=$qq}"&qq=reviewed+book

The results of the preceding query should look like this:

<?xml version="1.0" encoding="UTF-8"?> 
<response> 
<lst name="responseHeader"> 
<int name="status">0</int> 
<int name="QTime">2</int> 
<lst name="params"> 
<str name="fl">*,score</str> 
<str name="qq">book reviewed</str> 
<str name="q">book AND reviewed AND _query_:"{!dismax qf=title pf=title^10 v=$qq}"</str> 
</lst> 
</lst> 
<result name="response" numFound="3" start="0" maxScore="0.77966106"> 
<doc> 
<float name="score">0.77966106</float> 
<str name="id">2</str> 
<str name="title">Some book reviewed</str> 
</doc> 
<doc> 
<float name="score">0.07087828</float> 
<str name="id">1</str> 
<str name="title">Reviewed solrcook book</str> 
</doc> 
<doc> 
<float name="score">0.07087828</float> 
<str name="id">3</str> 
<str name="title">Another reviewed little book</str> 
</doc> 
</result> 
</response>

As you can see, we have used the same and simple index, let us skip its description and step into the next section.

Let us focus on the query. The q parameter is built of two parts connected together with AND operator. The first one reviewed+AND+book is just a usual query with a logical operator AND defined. In the second part, building the query starts with a strange looking expression, _query_. This expression tells Solr that another query should be made that will affect the results list. We then see the expression stating that Solr should use the dismax query parser (the !dismax part) along with the parameters that will be passed to the parser (qf and pf).

And thats it! We get to the search results which we had expected.

One of the advantages of Solr is the ability to group results on the basis of the field's contents. This ability to group results using Solr is defined as faceting which can help us in several tasks that we need to do in our everyday work. For instance, getting the number of documents with the same values in a field (such as the companies from the same city) through the ability of value and ranges grouping, to the autocomplete features based on faceting. In this section, I will show you how to handle some of the important and common tasks when using faceting.

<field name="id" type="string" indexed="true" stored="true" required="true" /> 
<field name="name" type="text" indexed="true" stored="true" /> 
<field name="city" type="string" indexed="true" stored="true" />

<add> 
<doc> 
<field name="id">1</field> 
<field name="name">Company 1</field> 
<field name="city">New York</field> 
</doc> 
<doc> 
<field name="id">2</field> 
<field name="name">Company 2</field> 
<field name="city">California</field> 
</doc> 
<doc> 
<field name="id">3</field> 
<field name="name">Company 3</field> 
<field name="city">New York</field> 
</doc> 
</add>

<?xml version="1.0" encoding="UTF-8"?> 
<response> 
<lst name="responseHeader"> 
<int name="status">0</int> 
<int name="QTime">1</int> 
<lst name="params"> 
<str name="facet">true</str> 
<str name="facet.field">city</str> 
<str name="q">name:company</str> 
</lst> 
</lst> 
<result name="response" numFound="3" start="0"> 
<doc> 
<str name="city">New York</str> 
<str name="id">1</str> 
<str name="name">Company 1</str> 
</doc> 
<doc> 
<str name="city">California</str> 
<str name="id">2</str> 
<str name="name">Company 2</str> 
</doc> 
<doc> 
<str name="city">New York</str> 
<str name="id">3</str> 
<str name="name">Company 3</str> 
</doc> 
</result> 
<lst name="facet_counts"> 
<lst name="facet_queries"/> 
<lst name="facet_fields"> 
<lst name="city"> 
<int name="New York">2</int> 
<int name="California">1</int> 
</lst> 
</lst> 
<lst name="facet_dates"/>
</lst> 
</response>

<field name="id" type="string" indexed="true" stored="true" required="true" /> 
<field name="name" type="text" indexed="true" stored=
  "true" /> 
<field name="price" type="float" indexed="true" stored="true" />

<add> 
<doc> 
<field name="id">1</field> 
<field name="name">Book 1</field> 
<field name="price">70</field> 
</doc> 
<doc> 
<field name="id">2</field> 
<field name="name">Book 2</field> 
<field name="price">100</field> 
</doc> 
<doc> 
<field name="id">3</field> 
<field name="name">Book 3</field> 
<field name="price">210.95</field> 
</doc> 
<doc> 
<field name="id">4</field> 
<field name="name">Book 4</field> 
<field name="price">99.90</field> 
</doc> 
</add>

<?xml version="1.0" encoding="UTF-8"?> 
<response> 
<lst name="responseHeader"> 
<int name="status">0</int> 
<int name="QTime">1</int> 
<lst name="params"> 
<str name="facet">true</str> 
<arr name="facet.query"> 
<str>price:[60 TO 100]</str> 
<str>price:[200 TO 250]</str> 
</arr> 
<str name="q">name:book</str> 
</lst> 
</lst> 
<result name="response" numFound="4" start="0"> 
<doc> 
<str name="id">1</str> 
<str name="name">Book 1</str> 
<float name="price">70.0</float> 
</doc> 
<doc> 
<str name="id">2</str> 
<str name="name">Book 2</str> 
<float name="price">100.0</float> 
</doc> 
<doc> 
<str name="id">3</str> 
<str name="name">Book 3</str> 
<float name="price">210.95</float> 
</doc> 
<doc> 
<str name="id">4</str>
<str name="name">Book 4</str> 
<float name="price">99.9</float> 
</doc> 
</result> 
<lst name="facet_counts"> 
<lst name="facet_queries"> 
<int name="price:[60 TO 100]">3</int> 
<int name="price:[200 TO 250]">1</int> 
</lst> 
<lst name="facet_fields"/> 
<lst name="facet_dates"/> 
</lst> 
</response>

<field name="id" type="string" indexed="true" stored="true" required="true" /> 
<field name="title" type="text" indexed="true" stored="true" /> 
<field name="title_autocomplete" type="lowercase" indexed="true" stored="true">

<copyField source="title" dest="title_autocomplete" />

<fieldType name="lowercase" class="solr.TextField"> 
<analyzer>
<tokenizer class="solr.KeywordTokenizerFactory"/> 
<filter class="solr.LowerCaseFilterFactory" /> 
</analyzer> 
</fieldType>

<add> 
<doc> 
<field name="id">1</field> 
<field name="title">Lucene or Solr ?</field> 
</doc> 
<doc> 
<field name="id">2</field> 
<field name="title">My Solr and the rest of the world</field> 
</doc> 
<doc> 
<field name="id">3</field> 
<field name="title">Solr recipes</field> 
</doc> 
<doc> 
<field name="id">4</field> 
<field name="title">Solr cookbook</field> 
</doc> 
</add>

<?xml version="1.0" encoding="UTF-8"?> 
<response> 
<lst name="responseHeader"> 
<int name="status">0</int> 
<int name="QTime">16</int> 
<lst name="params"> 
<str name="facet">true</str> 
<str name="q">*:*</str> 
<str name="facet.prefix">so</str> 
<str name="facet.field">title_autocomplete</str> 
        <str name="rows">0</str>
</lst> 
</lst> 
<result name="response" numFound="4" start="0"/> 
<lst name="facet_counts"> 
<lst name="facet_queries"/> 
<lst name="facet_fields"> 
<lst name="title_autocomplete"> 
<int name="solr cookbook">1</int> 
<int name="solr recipes">1</int> 
</lst> 
</lst> 
<lst name="facet_dates"/> 
</lst> 
</response>

Geomatics (also known as geospatial technology or geomatics engineering) is a discipline of gathering, storing, processing, and delivering geographic information, or spatial referenced information. This geographic information is based out of longitudes (vertical lines) and latitudes (horizontal lines) and can be effectively used in various ways and forms. For instance, you wish to store the location of your company when your company has multiple locations; or sorting the search results based on the distance from a point. To be more specific, geospatial is playing around with different co-ordinates throughout the globe.

In this section, we will talk about and understand how to:

<field name="id" type="string" indexed="true" stored="true" required="true" /> 
<field name="name" type="text" indexed="true" stored="true" /> 
<field name="location" type="point" indexed="true" stored="true" multiValued="true" /> 

<dynamicField name="*_d" type="double" indexed="true" stored="true"/>

<fieldType name="point" class="solr.PointType" dimension="2" subFieldSuffix="_d"/>

<add> 
<doc> 
<field name="id">1</field> 
<field name="name">company</field> 
<field name="location">10,10</field> 
<field name="location">30,30</field> 
</doc> 
</add>

java -jar post.jar geodata.xml

<?xml version="1.0" encoding="UTF-8"?> 
<response> 
<lst name="responseHeader"> 
<int name="status">0</int> 
<int name="QTime">3</int> 
<lst name="params">
<str name="q">location:10,10</str> 
</lst> 
</lst> 
<result name="response" numFound="1" start="0"> 
<doc> 
<str name="id">1</str> 
<arr name="location"> 
<str>10,10</str> 
<str>30,30</str> 
</arr> 
<arr name="location_0_d"> 
<double>10.0</double> 
<double>30.0</double> 
</arr> 
<arr name="location_1_d"> 
<double>10.0</double> 
<double>30.0</double> 
</arr> 
<str name="name">company</str> 
</doc> 
</result> 
</response>

<field name="id" type="string" indexed="true" stored="true" required="true" /> 
<field name="name" type="string" indexed="true" stored="true" /> 
<field name="x" type="float" indexed="true" stored="true" /> 
<field name="y" type="float" indexed="true" stored="true" />

<add> 
<doc> 
<field name="id">1</field> 
<field name="name">Company 1</field> 
<field name="x">56.4</field> 
<field name="y">40.2</field> 
</doc> 
<doc> 
<field name="id">2</field> 
<field name="name">Company 2</field> 
<field name="x">50.1</field> 
<field name="y">48.9</field> 
</doc> 
<doc> 
<field name="id">3</field> 
<field name="name">Company 3</field> 
<field name="x">23.18</field> 
<field name="y">39.1</field> 
</doc> 
</add>

<?xml version="1.0" encoding="UTF-8"?> 
<response> 
<lst name="responseHeader">
<int name="status">0</int> 
<int name="QTime">2</int> 
<lst name="params"> 
<str name="q">company</str> 
<str name="sort">dist(2,x,y,0,0) asc</str> 
</lst> 
</lst> 
<result name="response" numFound="3" start="0"> 
<doc> 
<str name="id">3</str> 
<str name="name">Company 3</str> 
<float name="x">23.18</float> 
<float name="y">39.1</float> 
</doc> 
<doc> 
<str name="id">1</str> 
<str name="name">Company 1</str> 
<float name="x">56.4</float> 
<float name="y">40.2</float> 
</doc> 
<doc> 
<str name="id">2</str> 
<str name="name">Company 2</str> 
<float name="x">50.1</float> 
<float name="y">48.9</float> 
</doc> 
</result> 
</response>

Distributed search in Solr is a concept of splitting an index into multiple shards, querying, and/or merging results across these shards. Imagine a situation where either the index is too huge to fit on a single system, or you have a query which takes too long to execute. How would you handle such situations? Don't worry! We have distributed search concept in Solr which is especially designed to handle such situations.

Let us consider the above stated scenario where you need to apply distributed search concept in order to overcome the huge index and/or query execution time concerns.

To overcome this situation, you need to distribute a request across ALL shards in a list using the shard parameter. Our request would follow this syntax:

host:port/base_url[,host:port/base_url]

A sharded request will go to the standard request handler (not necessarily the original); however we can override it using shards.qt. The following are the list of components that support distributed search:

On the contrary, distributed search has a list of limitations which are:

In this chapter, we have learned how to query your Solr based on different criteria such as field value, usage of extended dismax query parser, sorting your search results, phrase search, boosting and prioritizing your document in the search result, and nesting your queries. By now you must have also learned what faceted, Geospatial, and distributed searches are and how to play around with them, based on varied scenarios and conditions.

In the next chapter, we will learn different ways of monitoring Solr, performance metrics we should know, agent-based and agent-less health checks, and how to monitor Solr using monitoring tools like Opsview, New Relic, and SPM.