Exploring graphs using GraphFrames

In this section, we explore data, modeled as a graph, using Spark GraphFrames. The vertices and edges of the graph are stored as DataFrames, and Spark SQL and DataFrame-based queries are supported to operate on them. As DataFrames can support a variety of data sources, we can our input vertices edges information from relational tables, files (JSON, Parquet, Avro, and CSV), and so on.

The vertex DataFrame must contain a column called id which specifies unique IDs for each vertex. Similarly, the edges DataFrame must contain two columns named src (source vertex ID) and dst (destination vertex ID). Both the vertices and edges DataFrames can contain additional columns for the attributes.

GraphFrames exposes a concise language-integrated API that unifies graph analytics and relational queries. The system optimizes across the steps based on join plans and performing algebraic optimizations. Machine learning code, external data sources, and UDFs can be integrated...