Interactive programming using InteractiveRunner

The Python SDK lets us develop pipelines in Read-Evaluate-Print-Loop (REPL) fashion. This is especially useful for various data science tools, such as Python notebooks. This book focuses on the data engineering part, so we will not install the complete notebook. Instead, will use a command-line utility. This should be able to demonstrate the benefits of interactive programming.

We will run IPython for a better user experience by using the following command:

$ kubectl exec -it packt-beam-5686785d65-2ww5m -- /bin/bash -c "python3 \'which ipython3\'"

This will create an IPython console whose prompt looks like this:

Python 3.7.12 (default, Sep  8 2021, 01:20:16) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.27.0 -- An enhanced Interactive Python. Type '?' for help.
 
In [1]:

Now, we can start REPL coding. We have included a sample...