Scrapy shell

Scrapy shell is a command-line interface that helps to debug scripts without running the entire crawler. We have to provide a URL, and Scrapy shell will open up an interface to interact with objects that the spider handles in its callbacks, such as a response object.

How to do it...

We can go through some simple usage of Scrapy's interactive shell. The steps are as follows:

1. Open up a Terminal window and type the following command:

$ Scrapy shell http://books.toscrape.com/  

After loading the Scrapy shell, it will open up an interface to interact with the response object as follows:

2. We can use this interface to debug the selectors for the response object:

>>> response.xpath('//ol/li/article...