Putting It All Together

This book has so far introduced scraping techniques using a custom website, which helped us focus on learning particular skills. In this chapter, we will analyze a variety of real-world websites to show how the techniques we've learned in the book can be applied. First, we'll use Google to show a real-world search form, then Facebook for a JavaScript-dependent website and API, Gap for a typical online store, and finally, BMW for a map interface. Since these are live websites, there is a risk they will change by the time you read this. However, this is fine because the purpose of this chapter's examples is to show you how the techniques learned so far can be applied, rather than to show you how to scrape any particular website. If you choose to run an example, first check whether the website structure has changed since these examples were made and whether their current terms...