Dataset joining

In this section, we will cover dataset joining techniques. We will also discuss some of Spark's special features for data joining plus some data joining solutions made easy with Spark.

After this section, we will be able to join data for various machine learning needs.

Dataset joining and its tool – the Spark SQL

In preparing datasets for a machine learning project, we often need to combine data from multiple datasets. For relational tables, the task is to join tables through a primary and foreign key relationship.

Joining two or more datasets together sounds easy, but can be very challenging and time consuming. In SQL, SELECT is the most frequently used command. As an example, the following is a typical SQL code to perform a join:

SELECT column1, column2, …
FROM table1, table2
WHERE table1.joincolumn = table2.joincolumn
AND search_condition(s);

To work with the table joining tasks mentioned before, data scientists and machine learning professionals often utilize...