Data confidentiality in Hadoop
Hadoop is a distributed system. All distributed systems are interconnected via a network. Networks are vulnerable to malicious sniffing of data. Data at rest can also be read if they are not protected via encryption.
Data confidentiality for data at rest is delegated to the OS that hosts the DataNode. Most modern OSes provide encryption schemes to protect data on disks under their purview. In this section, we will look at confidentiality over the wire and how to enable encryption when data is in transit.
HTTPS and encrypted shuffle
Encrypted shuffle is a feature that facilitates data confidentiality in the shuffle process. To recap, the shuffle step is where data moves from the Map to the Reduce tasks in a MapReduce job life cycle. The movement of data occurs across machines through the network. The transport used to move this data across the network is HTTP.
HTTP, by itself, sends data in clear text, that is, in an unencrypted form. This can lead to information...
