Configuring Batch Retention

Configuring batch retention involves determining the duration for which you want to keep the information related to batch runs, such as when they ran, how much time they took, whether they completed successfully or with a failure, and how long the staging data used by the batch is kept on the disk.

Note

This section primarily focuses on the Configure batch retention concept of the DP-203: Data Engineering on Microsoft Azure exam.

A service that allows you to control batch retention is Azure Batch, which is an Azure service that can be used to perform large-scale parallel batch processing. It is typically used for high-performance computing applications such as image analysis, 3D rendering, genome sequencing, and optical character recognition.

However, you can create an Azure Batch activity in a pipeline and use it to run some part of the process for better control of the executing environment, including the possibility to specify the retention...