Using Selenium for web scraping

Selenium is used to test web applications. It is mostly used to perform browser automation using various programming language-based libraries and browser drivers. As we saw in a previous section, Exploring Selenium, we can navigate and locate elements in a page using Selenium and perform crawling and scraping-related activities.

Let's look at a few examples of scraping contents from web pages using Selenium.

Example 1 – scraping product information

In this example, we will continue using the search results obtained from foundProducts in the Exploring Selenium section.

We will extract some specific information from each individual product link found in foundProducts, listed as follows...