Storing metrics using MetricsRepository
Deequ allows us to store the metrics we calculate on a dataframe using MetricsRepository. Deequ provides facilities to create both in-memory and file-based repositories. File-based repositories support local filesystems, Simple Storage Service (S3), and Hadoop Distributed File System (HDFS). Persisting data quality metrics allow us to run analysis to see trends and spot any volatility in the data.
Creating an in-memory repository is simple, as the next example shows:
val inMemoryRepo = new InMemoryMetricsRepository()
Example 7.6
Similarly, we can create a file-based repository as follows:
val fileRepo = FileSystemMetricsRepository(sparkSession, filePath)
Example 7.7
The metrics for each run are stored using a key of type ResultKey. ResultKey is defined as a case class with the following signature:
case class ResultKey(dataSetDate: Long, tags: Map[String, String] = Map.empty)
Example 7.8
Here is an example key of...
