SBP 4 – implement data as a product!

Implementing your data as a product enables flows through your data engineered platform. SLAs, cataloging, versioning, corrections, lineage, time series, and quality metadata are all capabilities of the logical architecture! In this best practice, you will want to refocus the software engineers on the data product and not just the data input and output of a system. This means you (the data engineer) are responsible for helping the software engineering team see data as a product and then build software solutions that accommodate that need. The effort will be to encode semantics with data and create various knowledge-aware data flows. This results in more software development effort because many open source and third-party tools do not yet support that goal today. However, they will over time as support grows for this need. The requirement to curate data ready for ML/AI use cases will help drive this effort.

Organizations are constantly...