As explained earlier, it doesn't matter if you are using SQL, DataFrame, or Dataset, the Apache Spark SQL parser returns an abstract syntax tree. However, DataFrames or Datasets can be used as starting points. The result of all these methods is again a tree-based structure called ULEP. The following figure is an example of such an AST (abstract syntax tree) adding an attribute coming from a row in a table and two literals together:
So as you can see, at the bottom of the tree (those nodes are also called leaves since they don't have any further nodes connecting to them) we have two integer literals: one and two. On top we have an operation taking those two literals and adding them together. You should note that those literals could also be loaded from a persistent data store. The Add operation virtually turns into another literal (three...
