Web scraping using APIs

Technically, obtaining data from APIs is easy, and is also different from the web scraping scenarios. Most of the data is in JSON format, and there is no use of XPath, CSS Selector, or any other parsing libraries. Some data found using APIs might contain a lot of HTML code. To deal with this content, XPath and CSS Selector might be required.

Important note

In the examples in this section, we have tried to omit APIs that require an API key (walmartlabs.com, nasa.gov, nytimes.com, maps.googleapis.com) or user authentication tools. An API key is an authenticated value provided by service providers (after user registration) that identifies the user using the API. Freely available APIs are being used in these examples.

With the help of a few examples, we will collect data returned via some APIs. This collected data will then be exported as JSON and CSV files.

Example 1 – holidays from the US calendar

https://date.nager.at/ contains information...