Avoiding SPOFs

A SPOF in a data pipeline is a part of the system that, if it fails, will stop the entire system from working. SPOFs can severely impact the reliability and availability of your pipeline, leading to data processing delays, loss of data, and disruptions in downstream analytics. Avoiding SPOFs involves implementing redundancy and fault tolerance in your data pipeline design. Redundancy means having backup resources to take over if the primary resource fails. Fault tolerance involves designing the system to continue operation, even in a degraded state, when some part of the system fails.

Using the same logger instance as before, let’s add some redundancy to the extract() function of our demo data pipeline. To do this, we create two extract functions: extract_from_source1() and extract_from_source2(). Both functions import the same data source, but the second function is only run if the first function fails:

def extract():     try:
...