Avoiding patches of data darkness

There are different lenses with which to measure data quality. In simple terms, you want clean, complete, accurate, consistent, timely, and unbiased data. You want your stakeholders to trust the data so they can build more sophisticated data products. Multiple personas using different views of the data should not get contradictory data points, and at no point should false facts be made visible because compliance and audit will uncover it sooner or later.

There are some common problems that every organization dealing with big data grapples with that lead to compromises in data quality, namely failed production jobs, lack of schema enforcement, lack of data consistency, lost data, and compliance requirements such as the GDPR. Let's examine these problems in the context of a simple airline use case of showing flight delays and see how Delta's features help address data quality.

Addressing problems in flight status using Delta

This use...