Collecting Data by Scraping Web Pages

The process of gathering data from web pages and extracting information from them is known as web scraping. More than 85% of websites are created using HTML, that is, Hypertext Markup Language. A markup language is one in which text content is embedded between tags. Apart from this, additional information can be added by using attributes within tags. A web page can be considered to be a document written using HTML. Thus, we need to know the basics of HTML to scrape web pages effectively. The following figure depicts the contents that are included within an HTML tag:

Figure 4.1: Tags and attributes of HTML

As you can see in the preceding figure, we can easily identify different elements within an HTML tag. In the next section, we will walk through an exercise in which we'll extract tag-based information from HTML files.

Exercise 40: Extraction of Tag-Based Information from HTML Files

In this exercise, we will...