1. Introduction to the Course FREE CHAPTER

• What Is Web Scraping
• How Web Scraping Works
• Web Scraping with Scrapy

2. Scrapy Installation

• Scrapy Installation for Windows
• Scrapy Installation for Ubuntu (Linux)
• Creating Scrapy Project
• Project Walkthrough

3. Scrapy Spider

• Creating Spider
• Sending Request
• Getting the Response
• Scrapy CSS Selector
• Selecting All the Data
• Extracting Data
• Spider Overview

4. CSS Selectors

• CSS Selectors Versus XPath: How to Select Web Elements
• Tagname, Class, and Id Selectors
• Attribute Selectors

5. XPath

• XPath Expressions
• XPath Attribute Selectors
• XPath text( ) Function

6. Scrapy Shell

• What Is the Scrapy Shell and How to Use It?
• fetch( ) Response
• Shell Configuration

7. Scrapy Items

• Structuring Data into Scrapy Item
• Using Item in Spiders
• Define Input and Output Processors for Item Fields
• Loading Items with Scrapy ItemLoaders
• Items, Processors, and ItemLoaders Overview

8. Exporting Data

• Output Extracted Data in JSON, CSV, and XML Formats
• Overwrite Previous Output
• Appending Data to Previous Output

9. Scrapy Item Pipeline

• How to Use Scrapy Item Pipelines
• Saving Data Locally to Excel ( XLSX ) Files
• Enable Item Pipelines in Settings
• MongoDB (Account) Setup
• Saving Data to MongoDB

10. Pagination

• Extracting Links from href Attributes
• Send Request to the Next Page
• start_requests( ) Method

11. Following Links

• How to Follow Links
• How to Select Data Using Regular Expressions with Scrapy
• Setting Up Custom Callback Function
• Parse Product Details Page

12. Scraping Tables

• HTML Tables
• Selecting Tables Data
• Extract Data from HTML Tables

13. Logging into Websites

• Data Hidden with Logging Forms
• Inspecting HTML Forms and Website Activity with Dev Tools
• Logging into Websites with FormRequest
• CSRF Protected Login Forms
• Extract CSRF Values from Forms

14. Scraping JavaScript Rendered Websites

• What Are JavaScript Rendered/Dynamic Websites?
• scrapy-playwright Installation
• Setting Up Playwright in Scrapy Project
• Using Playwright to Render Websites
• Scraping Data from Dynamic Websites

15. Scrapy Playwright

• Playwright Overview
• Playwright Page Object
• Logging in with Playwright
• Dynamic Websites with Loading Screens
• Wait for Selector/Elements Using Page Coroutines
• Dynamic Websites with Infinite Scroll
• Taking Screenshot of Websites
• Rendering Websites to PDF

16. API Endpoints

• Identifying API Calls
• Requesting Data from API
• Extracting Data from API

17. Settings

• Scrapy Project Settings
• Robots Text
• Middleware
• Autothrottle Extension

18. User Agents and Proxies

• What Are User Agents?
• User Agents with Scrapy
• What Are Proxies?
• Proxies with Scrapy

19. Tips and Tricks

• Spider Arguments
• Standalone Spiders
• Scrapy Shell with bpython
• Scrapy Get Versus Extract Method
• Logging

20. Project 1: Champions League Table from ESPN.com

• Overview
• Website Visual Inspection
• Finding the Selectors
• Building the Spider: Extract Teams Data
• Building the Spider: Extract Teams Details

21. Project 2: Amazon Product Rank

• Overview
• Scraper Visualization
• Finding the Selectors
• Building the Spider

22. Project 3: Extending Scraper with GUI

• Scraper Application
• Building the GUI (Application Interface)
• Running the Spider from the Application