Learning how to aggregate, group, and join data
Another set of basic skills a data engineer needs is the ability to aggregate, group, and join data together. Let’s learn how to do this using Scala in Spark!
val numDirectorsByShow: DataFrame =
dfDirectorByShowSelectExpr
.groupBy($"show_id")
.agg(
count($"director").alias("num_director")
)
numDirectorsByShow.show(10, 0)
Here is the output:
+-------+------------+ |show_id|num_director| +-------+------------+ |s1 |1 | |s2 |1 | |s3 |1 | |s4 |1 &...
