Data validation with the Great Expectations library
Great Expectations is an open source Python library that facilitates data validation and documentation. It provides a framework for defining, managing, and executing data quality checks, making it easier to ensure data integrity and reliability throughout the data pipeline. Quality checks can be executed at different stages of the data life cycle, as shown in the following diagram:
Figure 3.11 – Quality checks at different stages of the data life cycle
Let’s discuss each of the touch points in the data life cycle where quality checks can be applied, as illustrated in the preceding figure:
- Data entry: During data entry or data collection, checks are conducted to ensure that the data is accurately captured and recorded. This can involve verifying the format, range, and type of data, as well as performing validation checks against predefined rules or standards.
- Data transformation...
