Scraping information from websites
Governments or jurisdictions around the world are increasingly embracing the importance of open data, which aims to increase citizen involvement and informed decision-making, and also aims to make policies more open to public scrutiny. Some examples of open data initiatives around the world include https://www.data.gov/ (United States of America), https://data.gov.uk/ (United Kingdom), and https://data.gov.hk/en/ (Hong Kong).
These data portals often provide an API for programmatic access of data. However, an API is not available for some datasets, hence we need to rely on good old web scraping techniques to extract information from websites.
Beautiful Soup (https://www.crummy.com/software/BeautifulSoup/) is an incredibly useful package for scraping information from websites. Basically, everything marked with an HTML tag can be scraped with this wonderful package. Scrapy is also a good package for web scraping, but it is more like a framework for writing...
