How data mesh monitoring differs
In a centralized analytical system, you have a few fixed subscriptions that manage resources across the end-to-end analytical cycle. The pipelines, storage system, data warehouse, and all the analytical frameworks are common to entire the system. The monitoring system, too, focuses only on this set of resources. Any change to the analytical system’s architecture is systematically added and changes are made to the monitoring system. These changes are few and far between.
A data mesh environment, depending on your data mesh strategy, can be very dynamic. New landing zones and data products can get added to the mesh more frequently than to the central system. Each data product will have a different set of tools and services for ingestion, analytics, and serving. Monitoring all the landing zones means ingesting diagnostics and logs from all the landing zones into the central data management zone and then analyzing all of them. The monitoring data...
