Leveraging advanced window functions
Window functions are a way to perform calculations on a specific group of records or a sliding subset of records. They can be used to perform cumulative calculations, get values from other records relative to the position of the current record being processed, and perform ranking calculations. They can also use aggregate functions such as count
, avg
, min
, and max
. Let’s take a look at them now:
val windowSpecRatingMonth = Â Â Window.partitionBy("rating", "month") Â Â Â Â Â Â Â Â .orderBy("year", "month")
Windows are created by defining a window specification. In the preceding example, we defined a window called windowSpecRatingMonth
by partitioning our records by rating and month and ordering the records in the partition by year and month:
val dfWindowedLagLead = dfNetflixUSMetrics   .withColumn(     "cast_per_director...