Implementing compression

In the previous chapters, we learned about the importance of data compression in data lakes. As the size of the data in data lakes grows, it becomes important that we store the data in a compressed format in order to save on cost. There are numerous ways in which we can implement compression in Azure. There are a lot of compression libraries available in the market and technically, all we need to do is write some scripts to call those libraries to compress data. But writing ad hoc scripts brings its own maintenance complexity, so let's look at some easy-to-maintain ways of implementing compression in Azure using Azure Synapse Pipelines. The same can be achieved using Azure Data Factory too. Azure Synapse Pipelines is just the same Azure Data Factory implementation within Synapse Analytics.

Compressing files using Synapse Pipelines or ADF

In the previous chapter, we learned about ADF. Like ADF, Synapse Pipelines can be used to create pipelines to...