Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Apache Solr for Indexing Data
Apache Solr for Indexing Data

Apache Solr for Indexing Data: Enhance your Solr indexing experience with advanced techniques and the built-in functionalities available in Apache Solr

eBook
AU$14.99 AU$42.99
Paperback
AU$53.99
Subscription
Free Trial
Renews at AU$24.99p/m

What do you get with eBook?

Product feature icon Instant access to your Digital eBook purchase
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
OR
Modal Close icon
Payment Processing...
tick Completed

Billing Address

Table of content icon View table of contents Preview book icon Preview Book

Apache Solr for Indexing Data

Chapter 1. Getting Started

We will start this chapter with a quick overview of Solr, followed by a section that helps you get Solr up and running. We will also cover some basic building blocks of the Solr architecture, its directory structure, and its configurations files. This chapter covers following topics:

  • Overview and installation of Solr
  • Running Solr
  • The Solr architecture and directory structure
  • Multicore Solr

Overview and installation of Solr

Solr is the one of the most popular open source enterprise search platforms from the Apache Lucene open source project. Its features include full text search, faceted search, highlighting, near-real-time indexing, dynamic clustering, rich document handling, and geospatial search. Solr is highly reliable and scalable. This is the reason Solr powers the search features of the world's largest Internet sites, for example, Netflix, TicketMaster, SourceForge, and so on (source: https://wiki.apache.org/solr/PublicServers).

Solr is written in Java and runs as a standalone full text search server with a REST-like API. You feed documents into it (which is called indexing) via XML, JSON, CSV, and binary over HTTP. You query it through HTTP GET and receive XML, JSON, CSV, and binary results.

Let's go through the installation process of Solr. This section describes how to install Solr on various operating systems such as Mac, Windows, and Linux. Let's go through each of them one by one.

Installing Solr in OS X (Mac)

The easiest way to install Solr on OS X is by using homebrew. If you are not aware of homebrew and don't have homebrew installed on your Mac, then go to http://brew.sh/. Homebrew is the easiest way of installing packages/software on Mac.

You will require JRE 1.7 or above to install Solr on OS X. Just type java –version in the terminal and see what the version of JRE installed in your computer is. If it's less than 1.7, then you need to upgrade it to higher version and proceed with the following instructions.

Just type the following command in the terminal and it will automatically download all the files needed for Solr. Sit back and relax for a few minutes until it completes:

$ brew install Solr

Running Solr

To test whether your installation was completed successfully, you need to run Solr. Type these commands in the terminal to run it:

$ cd /usr/local/Cellar/solr/4.4.0/libexec/example/
$ java -jar start.jar

After you run the preceding commands, you will see lots of dumping messages/logs on the terminal. Don't worry! It's normal. Just try to fix any error if it is there. Once the messages are stopped and there is no error message, simply go to any web browser and type http://localhost:8983/solr/#/.

Tip

Downloading the example code

You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

You will see following screen on your browser:

Running Solr

Fresh Solr do not contain any data. In Solr terminology, data is termed as a document. You will learn how to index data in Solr in upcoming chapters.

Installing Solr in Windows

There are multiple ways of installing Solr on a Windows machine. Here, I have explained the way to set up Solr with Jetty running as a service via NSSM:

  1. Install the latest Java JDK from http://www.oracle.com/technetwork/java/javase/downloads/index.html.
  2. Download the latest Solr release (ZIP version) from http://www.apache.org/dyn/closer.cgi/lucene/solr/. At the time of writing this book, the latest Solr release was 4.10.1.
  3. Unzip the Solr download. You should have files as shown in the following screenshot. Open the example folder.
    Installing Solr in Windows
  4. Copy the etc, lib, logs, solr, and webapps folders and start.jar to C:\solr (you will need to create the folder at C:\solr), as shown in the following screenshot:
    Installing Solr in Windows
  5. Now open the C:\solr\solr folder and copy the contents back to the root C:\solr folder. When you are done, you can delete the C:\solr\solr folder. See the following image, the selected folder you can delete now:
    Installing Solr in Windows

    At this point, your C:\solr directory should look like what is shown in the following screenshot:

    Installing Solr in Windows
  6. Solr can be run at this point if you start it from the command line. Change your directory to C:\solr and then run java -Dsolr.solr.home=C:/solr/ -jar start.jar.
  7. If you go to http://localhost:8983/solr/, you should see the Solr dashboard.
  8. Now Solr is up and running, so we can work on getting Jetty to run as a Windows service. Since Jetty comes bundled with Solr, all that we need to do is run it as a service. There are several options to do this, but the one I prefer is through Non-Sucking Service Manager (NSSM)program in windows which is the, the most compatible service manager across Windows environment. NSSM can be downloaded from http://nssm.cc/download.
  9. Once you have downloaded NSSM, open the win32 or win64 folder as appropriate and copy nssm.exe to your C:\solr folder.
  10. Open Command Prompt, change the directory to C:\solr, and then run nssm install Solr.
  11. A dialog will open. Select java.exe as the application located at C:\Windows\System32\.
  12. In the options input box, enter: Dsolr.solr.home=C:/solr/ -Djetty.home=C:/solr/ -Djetty.logs=C:/solr/logs/ -cp C:/solr/lib/*.jar;C:/solr/start.jar -jar C:/solr/start.jar.
  13. Click on Install service. You should get a service successfully installed message.
  14. Finally run net start Solr.
  15. Jetty should now be running as a service. Check this by going to http://localhost:/8983/solr/.

Installing Solr on Linux

To install Solr on Linux/Unix, you will need Java Runtime Environment (JRE) version 1.7 or higher. Then follow these steps:

  1. Download the latest Solr release (.tgz) from http://www.apache.org/dyn/closer.cgi/lucene/solr/. At the time of writing this book, the latest release was 4.10.1.
  2. Unpack the file to your desired location.
  3. Solr runs inside a Java servlet container, such as Tomcat, Jetty, and so on. Solr distribution includes a working demo server in the example directory, which runs in Jetty. You can use Jetty servlet container, or use your preferred servlet container. If you are using a servlet container other than Jetty and it's already running, then stop that server.
  4. Copy the solr-4.10.1.war file from the Solr distribution under the dist directory to the webapps directory of your servlet container. Change the name of this file; it must be named solr.war.
  5. Copy the Solr home directory, solr-4.x.0/example/solr/, from the distribution to your desired Solr home location.
  6. Start your servlet container, passing to it the location of your Solr home in one of these ways:
    1. Set the solr.solr.home Java system property to your Solr home (for example, using this example jetty setup: java -Dsolr.solr.home=/some/dir -jar start.jar).
    2. Configure the servlet container so that a JNDI lookup of java:comp/env/solr/home by the Solr web app will point to your Solr Home.
    3. Start the servlet container in the directory containing ./solr. The default Solr Home is solr under the JVM's current working directory ($CWD/solr).
  7. To confirm the installation, just go to http://localhost:/8983/solr/ and you will see the Solr dashboard. Now your Solr is up and running.

Thus, by the end of the installation, your Solr is up and running. But since we have not fed any data into Solr, it will not index any data. Let's try to insert some example data into our server.

The Solr download comes with example data bundled in it. We can use the same data for indexing as an example. Go to the exampledocs directory under the example directory. Here, you will see a lot of files. Now go to the command line (terminal) and type the following commands:

$ cd $SOLR_HOME/example/exampledocs/
$ ./post.sh vidcard.xml

Within the post.sh file, the script will call http://localhost:8983/solr/update using curl to post xml data from the vidcard.xml file. When the import completes (without any error), you will see a message that looks something like this:

Installing Solr on Linux

Now let's try to check out our imported data from web browser. Try http://localhost:8983/solr/select?q=*:*&wt=json to fetch all of the data in your Solr instance, like this:

Installing Solr on Linux

When you see the preceding data, it means that your Solr server is running properly and is ready to index your desired feed. You will be reading indexing in depth in upcoming chapters.

The Solr architecture and directory structure

In real-world scenarios, Solr runs with other applications on a web server. A typical example is an online store application. The store provides a user interface, a shopping cart, an items catalogue, and a way to make purchases. It needs to store this information some sort of database. Here, Solr makes easy so add the capability of searching data in the online store. To make data searchable, you need to feed it to Solr for indexing. Data can be fed to Solr in various ways and also in various formats, such as .pdf, .doc, .txt, and so on. In the process of feeding data to Solr, you need to define a schema. A schema is a way of telling Solr about data and how you want to make your data indexed. A lot many factors need to be considered while feeding data, which we will discuss in detail in upcoming chapters.

Solr queries are RESTful, which means that a Solr query is just a simple HTTP request and the response is a structured document, mainly in XML, but it could be JSON, CSV, or any other format as well based on your requirement. A typical architecture of Solr in the real world looks something like this:

The Solr architecture and directory structure

Do not worry if you are not able to understand the preceding diagram right now. We will cover every component related to indexing in detail. The purpose of this diagram is to give you a feel of the current architecture of Solr and its working in the real world. If you see the preceding diagram properly, you will find two .xml files named schema.xml and solrconfig.xml. These are the two most important files in the Solr configuration and are considered the building blocks of Solr.

Solr directory structure

Here's the directory layout of a typical Solr Home directory:

| + conf 
|     - schema.xml 
|     - solrconfig.xml 
|     - stopwords.txt
|     - synonyms.txt etc
| + data 
|     - index 
|     - spellchecker

Let's get a brief understanding of solrconfig.xml and schema.xml here before we proceed further, as these are the building blocks of Solr (as stated earlier). We will cover them in detail in the next few chapters.

The solrconfig.xml file is the core configuration file of Solr, with most parameters affecting Solr itself directly. This file can be found in the solr/collection1/conf/ directory. When configuring Solr, you'll work with solrconfig.xml often. The file consists of a series of XML statements that set configuration values, and some of the most important configurations are:

  • Defining data dir (the directory where indexed files remain)
  • Request handlers (handle upcoming HTTP requests)
  • Listeners
  • Request dispatchers (used to manage HTTP communications)
  • Admin web interface settings
  • Replication and duplication parameters

These are some of the important configurations defined in solrconfig.xml. This file is well commented; I would advise you to go through it from the start and read all the comments. You will get a very good understanding of the various components involved in the Solr configuration.

The second most important configuration file is called schema.xml. This file can be found in the solr/collection1/conf/ directory. As the name says, this file is used to define the schema of the data (content) that you want to index and make searchable. Data is called document in Solr terminology. The schema.xml file contains all the details about the fields that your documents can contain, and how these fields should be dealt with when adding documents to the index or when querying those fields. This file can be divided broadly into two sections:

  • The types section (the definitions of all types)
  • The fields section (the definitions of the document structure using types)

The structure of your document should be defined as a field under the fields section. Let's say you have to define a book as a document in Solr with fields as isbn, title, author, and price. The schema will be as follows:

<field name="isbn" type="string" required="true" indexed="true" stored="true"/> <field name="title" type="text_general" indexed="true" stored="true"/>
<field name="author" type="text-general" indexed="true" stored="true" multiValued="true"/>
<field name="price" type="int" indexed="true" stored="true"/>

In the preceding schema, you see a type attribute, which defines the data type of the field. You can change the behavior of the field by changing the type. The multiValued attribute is used to tell Solr that the field can hold multiple values, while the required attribute makes the field mandatory for creating a document. After the fields section ends, we need to mention which field is going to be unique. In our case, it is going to be isbn:

<uniqueKey>isbn</uniqueKey>

The schema.xml file is also well-commented file. I will again advise you to go through the comments of this file, for starting this will help you understand the various field types and data types in detail.

Cores in Solr (Multicore Solr)

Solr cores make it possible to run multiple indexes with different configurations and schemas in a single Solr instance. The multicore feature of Solr helps in unified administration of Solr instances for complete and different applications. Cores in Solr are fairly isolated and have their own configuration and schema files. This helps manage cores at runtime (create or remove) from a Solr instance without restarting the process.

Cores in Solr are managed through a configuration file called solr.xml. The solr.xml file is present in your Solr Home directory. Since its inception, solr.xml has evolved from configuring one core to managing multiple cores and eventually defining parameters for SolrCloud. Do not worry much about SolrCloud if you are not aware of it, as we have a dedicated chapter that covers SolrCloud in detail. In brief, SolrCloud is a terminology used in distributed search and indexing. When we need to index huge amounts of data, we need to think of scalability and performance. This is where SolrCloud comes into the picture.

Starting from Solr 4.3, Solr will maintain two distinct formats for solr.xml; one is legacy and the other is discovery mode. The legacy format will be supported until the 4.x.0 series and it will be deprecated in the 5.0 release of Solr. The default solr.xml config file looks something like this:

<solr>

  <solrcloud>
    <str name="host">${host:}</str>
    <int name="hostPort">${jetty.port:8983}</int>
    <str name="hostContext">${hostContext:solr}</str>
    <int name="zkClientTimeout">${zkClientTimeout:30000}</int>
    <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
  </solrcloud>

  <shardHandlerFactory name="shardHandlerFactory"
    class="HttpShardHandlerFactory">
    <int name="socketTimeout">${socketTimeout:0}</int>
    <int name="connTimeout">${connTimeout:0}</int>
  </shardHandlerFactory>

</solr>

The preceding configuration shows that Solr configurations are SolrCloud friendly, but this does not mean that Solr is running in SolrCloud mode, unless you start Solr with some special parameters (explained in the SolrCloud Chapter 10, Distributed Indexing). To configure multiple cores in Solr in legacy format, you need to edit the solr.xml file with the following code snippet and remove the existing discovery code from solr.xml:

<solr persistent="false">
    <cores adminPath="/admin/cores" defaultCoreName="core1">
    <core name="core1" instanceDir="core1" />
    <core name="core2" instanceDir="core2" />
  </cores>
</solr>

Now you need to create two cores (new directories, core1 and core2) in the Solr directory. You also need to create Solr configuration files for new cores. To do this, just copy the same configuration files (the conf directory in collections1) in both cores for now and restart the Solr server after you have made these settings.

Once you restart the Solr server with the preceding configuration, two cores will be created, with names core1 and core2 and the existing default Solr configuration settings. The instanceDir variable defines the directory name relative to solr.xml—where to look for configuration and data files. You can modify the paths of these cores according to your wishes and the configuration files according to your use case. You can also change the names of the cores.

You can verify your settings by opening the following URL in your browser: http://localhost:8983/solr/.

You will see two new cores created in the Solr dashboard. Currently, there is no document in any of the cores because we have not indexed any data so far. So, this concludes the process of creating multiple cores in Solr.

Summary

Thus, by the end of the first chapter, you have learned what Solr is, how to install and run it on various operating systems, what the various components and basic building blocks of Solr are (such as its configuration files and directory structure), and how to set up configuration files. You also learned in brief about the architecture of Solr. In the last section, we covered multicore setup in the Solr 4.x.0 series. However, the legacy method of multicore setup is going to be deprecated in the Solr 5.x release and then it's going to be only discovery mode, which is called SolrCloud.

In the next chapter, we will look deeply into the various components used in Solr configuration files, such as tokenizers, analyzers, filters, field types, and so on.

Left arrow icon Right arrow icon

Key benefits

  • Learn about distributed indexing and real-time optimization to change index data on fly
  • Index data from various sources and web crawlers using built-in analyzers and tokenizers
  • This step-by-step guide is packed with real-life examples on indexing data

Description

Apache Solr is a widely used, open source enterprise search server that delivers powerful indexing and searching features. These features help fetch relevant information from various sources and documentation. Solr also combines with other open source tools such as Apache Tika and Apache Nutch to provide more powerful features. This fast-paced guide starts by helping you set up Solr and get acquainted with its basic building blocks, to give you a better understanding of Solr indexing. You’ll quickly move on to indexing text and boosting the indexing time. Next, you’ll focus on basic indexing techniques, various index handlers designed to modify documents, and indexing a structured data source through Data Import Handler. Moving on, you will learn techniques to perform real-time indexing and atomic updates, as well as more advanced indexing techniques such as de-duplication. Later on, we’ll help you set up a cluster of Solr servers that combine fault tolerance and high availability. You will also gain insights into working scenarios of different aspects of Solr and how to use Solr with e-commerce data. By the end of the book, you will be competent and confident working with indexing and will have a good knowledge base to efficiently program elements.

Who is this book for?

This book is for developers who want to increase their experience of indexing in Solr by learning about the various index handlers, analyzers, and methods available in Solr. Beginner level Solr development skills are expected.

What you will learn

  • Get to know the basic features of Solr indexing and the analyzers/tokenizers available
  • Index XML/JSON data in Solr using the HTTP Post tool and CURL command
  • Work with Data Import Handler to index data from a database
  • Use Apache Tika with Solr to index word documents, PDFs, and much more
  • Utilize Apache Nutch and Solr integration to index crawled data from web pages
  • Update indexes in real-time data feeds
  • Discover techniques to index multi-language and distributed data in Solr
  • Combine the various indexing techniques into a real-life working example of an online shopping web application

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : Dec 28, 2015
Length: 160 pages
Edition : 1st
Language : English
ISBN-13 : 9781783553242
Category :
Concepts :
Tools :

What do you get with eBook?

Product feature icon Instant access to your Digital eBook purchase
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
OR
Modal Close icon
Payment Processing...
tick Completed

Billing Address

Product Details

Publication date : Dec 28, 2015
Length: 160 pages
Edition : 1st
Language : English
ISBN-13 : 9781783553242
Category :
Concepts :
Tools :

Packt Subscriptions

See our plans and pricing
Modal Close icon
AU$24.99 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
AU$249.99 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just AU$5 each
Feature tick icon Exclusive print discounts
AU$349.99 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just AU$5 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total AU$ 205.97
Apache Solr for Indexing Data
AU$53.99
Apache Solr Search Patterns
AU$75.99
Solr Cookbook - Third Edition
AU$75.99
Total AU$ 205.97 Stars icon
Banner background image

Table of Contents

12 Chapters
1. Getting Started Chevron down icon Chevron up icon
2. Understanding Analyzers, Tokenizers, and Filters Chevron down icon Chevron up icon
3. Indexing Data Chevron down icon Chevron up icon
4. Indexing Data – The Basic Technique and Using Index Handlers Chevron down icon Chevron up icon
5. Indexing Data with the Help of Structured Datasources – Using DIH Chevron down icon Chevron up icon
6. Indexing Data Using Apache Tika Chevron down icon Chevron up icon
7. Apache Nutch Chevron down icon Chevron up icon
8. Commits, Real-Time Index Optimizations, and Atomic Updates Chevron down icon Chevron up icon
9. Advanced Topics – Multilanguage, Deduplication, and Others Chevron down icon Chevron up icon
10. Distributed Indexing Chevron down icon Chevron up icon
11. Case Study of Using Solr in E-Commerce Chevron down icon Chevron up icon
Index Chevron down icon Chevron up icon

Customer reviews

Rating distribution
Full star icon Empty star icon Empty star icon Empty star icon Empty star icon 1
(1 Ratings)
5 star 0%
4 star 0%
3 star 0%
2 star 0%
1 star 100%
Sanjay Tiwari Nov 28, 2018
Full star icon Empty star icon Empty star icon Empty star icon Empty star icon 1
Book is good but very difficult to get sample code, would advise not to buy any book from packt publication.
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

How do I buy and download an eBook? Chevron down icon Chevron up icon

Where there is an eBook version of a title available, you can buy it from the book details for that title. Add either the standalone eBook or the eBook and print book bundle to your shopping cart. Your eBook will show in your cart as a product on its own. After completing checkout and payment in the normal way, you will receive your receipt on the screen containing a link to a personalised PDF download file. This link will remain active for 30 days. You can download backup copies of the file by logging in to your account at any time.

If you already have Adobe reader installed, then clicking on the link will download and open the PDF file directly. If you don't, then save the PDF file on your machine and download the Reader to view it.

Please Note: Packt eBooks are non-returnable and non-refundable.

Packt eBook and Licensing When you buy an eBook from Packt Publishing, completing your purchase means you accept the terms of our licence agreement. Please read the full text of the agreement. In it we have tried to balance the need for the ebook to be usable for you the reader with our needs to protect the rights of us as Publishers and of our authors. In summary, the agreement says:

  • You may make copies of your eBook for your own use onto any machine
  • You may not pass copies of the eBook on to anyone else
How can I make a purchase on your website? Chevron down icon Chevron up icon

If you want to purchase a video course, eBook or Bundle (Print+eBook) please follow below steps:

  1. Register on our website using your email address and the password.
  2. Search for the title by name or ISBN using the search option.
  3. Select the title you want to purchase.
  4. Choose the format you wish to purchase the title in; if you order the Print Book, you get a free eBook copy of the same title. 
  5. Proceed with the checkout process (payment to be made using Credit Card, Debit Cart, or PayPal)
Where can I access support around an eBook? Chevron down icon Chevron up icon
  • If you experience a problem with using or installing Adobe Reader, the contact Adobe directly.
  • To view the errata for the book, see www.packtpub.com/support and view the pages for the title you have.
  • To view your account details or to download a new copy of the book go to www.packtpub.com/account
  • To contact us directly if a problem is not resolved, use www.packtpub.com/contact-us
What eBook formats do Packt support? Chevron down icon Chevron up icon

Our eBooks are currently available in a variety of formats such as PDF and ePubs. In the future, this may well change with trends and development in technology, but please note that our PDFs are not Adobe eBook Reader format, which has greater restrictions on security.

You will need to use Adobe Reader v9 or later in order to read Packt's PDF eBooks.

What are the benefits of eBooks? Chevron down icon Chevron up icon
  • You can get the information you need immediately
  • You can easily take them with you on a laptop
  • You can download them an unlimited number of times
  • You can print them out
  • They are copy-paste enabled
  • They are searchable
  • There is no password protection
  • They are lower price than print
  • They save resources and space
What is an eBook? Chevron down icon Chevron up icon

Packt eBooks are a complete electronic version of the print edition, available in PDF and ePub formats. Every piece of content down to the page numbering is the same. Because we save the costs of printing and shipping the book to you, we are able to offer eBooks at a lower cost than print editions.

When you have purchased an eBook, simply login to your account and click on the link in Your Download Area. We recommend you saving the file to your hard drive before opening it.

For optimal viewing of our eBooks, we recommend you download and install the free Adobe Reader version 9.