In any enterprise grade Kafka deployment, you need to build a solid governance framework to ensure security of confidential data along with who is dealing with data and what kind of operations are performed on data. Moreover, governance framework ensures who can access what data and who can perform operations on data elements. There are tools available such as Apache Atlas and Apache Ranger, which will help you define a proper governance framework around Kafka.
The fundamental data element in Kafka is Topic. You should define all your governance processes around Topic data element.
The following diagram represents how data governance can be applied in Kafka using Apache Atlas and Ranger:
To give an overview of the diagram, we can sum up all the steps as follows:
- Create tags in Apache Atlas. Each tag corresponds to a...
