Parsing HTML

Downloading raw text or a binary file is a good starting point, but the main language of the web is HTML.

HTML is a structured language, defining different parts of a document such as headers and paragraphs. HTML is also hierarchical, defining sub-elements. The ability to parse raw text into a structured document is basically to be able to extract information automatically from a web page. For example, some text can be relevant if enclosed in a particular class div or after a header h3 tag.

Getting ready

We'll use the excellent Beautiful Soup module to parse the HTML text into a memory object that can be analyzed. We need to use the beautifulsoup4 package to use the latest Python 3 version that is available...