In Chapter 2, Interacting with Web Applications, we learned how to interact with a web application programmatically using Python and the requests library. In this chapter, we will cover the following topics:
- Web application mapping
- Creating our own crawler/spider with Scrapy
- Making our crawler recursive
- Scraping interesting stuff