Trade-offs in designing streaming architectures
Spark has multiple ways of achieving the end goal with tunable performance, cost, and quality. Hence, there is a need for a process to understand the goals/requirements of use cases. For each goal, define the strategy needed; for each goal, define the resources required; compute the cost of resources employed. The process is repeated until expectations are balanced. This is where trade-offs need to be considered, as shown in the following diagram; either the goal or resource has to be tweaked:
Figure 4.14 – Balancing streaming service requirements and resources
It is a balancing act of managing the various goals with the resources that the team is willing to bring to the table. Goals refer to requirements regarding scalability, performance, processing logic, quality, reliability, and availability. Resources refer to compute, storage, and integration services, and the effort needed to not only create but...
