Transforming and cleaning data

After our data has been loaded into a Spark dataframe, we can manipulate it in different ways. We can directly manipulate our Spark dataframe or save the data to a table, and use Structured Query Language (SQL) statements to perform queries, data definition language (DDL), data manipulation language (DML), and more.

You will need to have the Voting_Turnout_US_2020 dataset loaded into a Spark dataframe.

Spark data frames

A Spark data frame is a tabular collection of data organized in rows with named columns, which in turn have their own data types. All this information is stored as metadata that we can access using displaySchema in order to display the data types of each column or display the actual data, or describe in order to view the statistical summary of the data. One of its characteristics is that it is able to handle big amounts of data thanks to its distributed nature.

We can perform transformations such as selecting rows and columns...