The first application in this book is going to be a web scraping application that will scrape weather forecast information from https://weather.com and present it in a terminal. We will add some options that can be passed as arguments to the application, such as:
- The temperature unit (Celsius or Fahrenheit)
- The area where you can get the weather forecast
- Output options where the user of our application can choose between the current forecast, a five-day forecast, a ten-day forecast, and the weekend
- Ways to complement the output with extra information such as wind and humidity
Apart from the aforementioned arguments, this application will be designed to be extendable, which means that we can create parsers for different websites to get a weather forecast, and these parsers will be available as argument options.
In this chapter, you will learn how to:
- Use object-oriented programming concepts in Python applications
- Scrape data from websites using the BeautifulSoup package
- Receive command line arguments
- Utilize the inspect module
- Load Python modules dynamically
- Use Python comprehensions
- Use Selenium to request a webpage and inspect its DOM elements
Before we get started, it is important to say that when developing web scraping applications, you should keep in mind that these types of applications are susceptible to changes. If the developers of the site that you are getting data from change a CSS class name, or the structure of the HTML DOM, the application will stop working. Also, if the URL of the site we are getting the data from changes, the application will not be able to send requests.Â
