Implementing data observability in a project

In this section, we will learn how to implement data observability at scale in an organization, using the technical roadmap presented in the first part of this chapter.

Here, we will provide a macro view, at the organization level, of how observability will be progressively adopted. For the micro level, which includes inside a project, pipeline, or data product, please refer to Part 2 – Implementing Data Observability.

We will consider a company called PetCie, a pet store company that’s active in five countries and produces and sells dog food to resellers.

This company currently has three projects:

• Sales reporting: A financial dashboard that’s used by the CFO so that they can follow the day-to-day sales to the resellers. This report is the outcome of a Tableau dashboard that’s linked to a PostgreSQL table and fed by a Spark application. They plan on changing Tableau into a Qlik process this quarter...