Chapter 6: Using Your Preferred Language with Portability

In the previous chapters, we focused on the Java SDK – or various Java SDK-based DSLs – but what if we want to implement our data transformation logic in a completely different language, such as Python or Go? One of the main goals of Apache Beam is portability. We have already seen the portability of pipelines between different Runners and between batch and streaming semantics. In this chapter, we will explore the last aspect of portability – portability between SDKs.

We will outline how the portability layer works (Apache Beam often calls it the Fn API – pronounced Fun API) so that the result is portable. The desired goal is to enable Runners so that they don't have to understand the SDK (the language we want to use to implement our pipeline), yet can still execute it successfully. That way, new SDKs can be created without us needing to make modifications to the currently existing Runners...