You will need pandas, NumPy, and Matplotlib to complete the recipes in this chapter. I used pandas 2.1.4, but the code will run on pandas 1.5.3 or later.

The code in this chapter can be downloaded from the book’s GitHub repository, https://github.com/PacktPublishing/Python-Data-Cleaning-Cookbook-Second-Edition.

JavaScript Object Notation (JSON) has turned out to be an incredibly useful standard for transferring data from one machine, process, or node to another. Often, a client sends a data request to a server, upon which that server queries the data in local storage and then converts it from something like an SQL Server, MySQL, or PostgreSQL table or tables into JSON, which the client can consume. This is sometimes complicated further by the first server (say, a web server) forwarding the request to a database server. JSON facilitates this, as does XML, by doing the following:

• Being readable by humans
• Being consumable by most client devices
• Not being limited in structure

JSON is quite flexible, which means that it can accommodate just about anything, no matter how unwise. The structure can even change within a JSON file, so different keys might be present at different points. For example, the file might begin with some explanatory...

In the previous recipe, we discussed one significant advantage (and challenge) of working with JSON data – its flexibility. A JSON file can have just about any structure its authors can imagine. This often means that this data does not have the tabular structure of the data sources we have discussed so far and that pandas DataFrames have. Often, analysts and application developers use JSON precisely because it does not insist on a tabular structure. I know I do!

Retrieving data from multiple tables often requires us to do a one-to-many merge. Saving that data to one table or file means duplicating data on the “one” side of the one-to-many relationship. For example, student demographic data is merged with data on the courses studied, and the demographic data is repeated for each course. With JSON, duplication is not required to capture these items of data in one file. We can have data on the courses studied nested...

We use Beautiful Soup in this recipe to scrape data from a web page and load that data into pandas. Web scraping is very useful when there is data on a website that is updated regularly but there is no API. We can rerun our code to generate new data whenever the page is updated.

Unfortunately, the web scrapers we build can be broken when the structure of the targeted page changes. That is less likely to happen with APIs because they are designed for data exchange and carefully curated with that end in mind. The priority for most web designers is the quality of the display of information, not the reliability and ease of data exchange. This causes data cleaning challenges that are unique to web scraping, including HTML elements that house the data in surprising and changing locations, formatting tags that obfuscate the underlying data, and explanatory text that aid data interpretation being difficult to retrieve. In addition to these challenges, scraping...

When working with large datasets, we sometimes need to rely on distributed resources to clean and manipulate our data. With Apache Spark, analysts can take advantage of the combined processing power of many machines. We will use PySpark, a Python API for working with Spark, in this recipe. We will also go over how to use PySpark tools to take a first look at our data, select parts of our data, and generate some simple summary statistics.

Getting ready

To run the code in this section, you need to get Spark running on your computer. If you have installed Anaconda, you can follow these steps to work with Spark:

1. Install Java with conda install openjdk.
2. Install PySpark with conda install pyspark or conda install -c conda forge pyspark.
3. Install findspark with conda install -c conda-forge findspark.

   Note

   Installation of PySpark can be tricky, particularly setting the necessary environment variables. While findspark...

There are several reasons why we might want to serialize a JSON file:

• We may have retrieved the data with an API but need to keep a snapshot of the data.
• The data in the JSON file is relatively static and informs our data cleaning and analysis over multiple phases of a project.
• We might decide that the flexibility of a schema-less format such as JSON helps us solve many data cleaning and analysis problems.

It is worth highlighting this last reason to use JSON – that it can solve many data problems. Although tabular data structures clearly have many benefits, particularly for operational data, they are often not the best way to store data for analysis purposes. In preparing data for analysis, a substantial amount of time is spent either merging data from different tables or dealing with data redundancy when working with flat files. Not only are these processes time-consuming but every merge or reshaping leaves the door open...

There may be times when we want to persist data without overwriting a prior version of the data file. This can be accomplished by appending a time stamp to a filename or a unique identifier. However, there are more elegant solutions available. One such solution is the Delta Lake library, which we will explore in this recipe.

We will work with the land temperature data again in this recipe. We will load the data, save it to a data lake, and then save an altered version to the same data lake.

Getting ready

We will be using the Delta Lake library in this recipe, which can be installed with pip install deltalake. We will also need the os library so that we can make a directory for the data lake.

How to do it...

You can get started with the data and version it as follows:

1. We start by importing the Delta Lake library. We also create a folder called temps_lake for our data versions:

   import pandas as pd
   from deltalake.writer import write_deltalake...

The recipes in this chapter examined importing and data preparation of non-tabular data in a variety of forms, including JSON and HTML. We introduced Spark for working with big data and discussed how to persist tabular and non-tabular data. We also examined how to create a data lake for versioning. We will learn how to take the measure of our data in the next chapter.

Join our community on Discord

Join our community’s Discord space for discussions with the author and other readers:

https://discord.gg/p8uSgEAETX