Discussing the Photon product roadmap
Photon is a brand-new query engine. As we saw in the previous section, it must implement vectorized kernels for all operations that are possible in Apache Spark. At the time of writing, there are still a few operations that are not available with Photon:
- Photon Scan and Write operators work with Delta and Parquet files only
- Window and Sort operations are not yet supported
- User-defined functions are not yet supported
- Spark Structured Streaming is not yet supported
Work is in progress on all of these and I recommend that you visit https://docs.databricks.com/runtime/photon.html#limitations for the latest status.
That said, does this mean that you cannot use Window and Sort operations with Databricks SQL or that you cannot use your own user-defined functions in Databricks SQL?
You can! Recall Figure 9.3. It shows that the tasks scheduled by Spark are executed by Photon or JVM-Core as applicable. If an operation is...
