Determining the most popular non-stop flights

Expanding upon our tripGraph GraphFrame, the following query will allow us to find the most popular non-stop flights in the US (for this dataset):

# Determine the most popular non-stop flights
import pyspark.sql.functions as func
topTrips = tripGraph \
  .edges \
  .groupBy("src", "dst") \
  .agg(func.count("delay").alias("trips"))

# Show the top 20 most popular flights (single city hops)
display(topTrips.orderBy(topTrips.trips.desc()).limit(20))

Note, while we are using the delay column, we're just actually doing a count of the number of trips. Here's the output:

As can be observed from this query, the two most frequent non-stop flights are between LAX (Los Angeles) and SFO (San Francisco). The fact that these flights are so frequent indicates their importance in the airline market. As noted in the New York Times article from April 4, 2016, Alaska Air Sees Virgin America as Key to West Coast (http://nyti.ms/2ea1uZR), acquiring slots at these two...