Best practices for overcoming common challenges

This section contains pointers and best practices that companies have developed over time to address many of the challenges discussed in the previous section.

Finding and gathering relevant data

We discussed data silos being a common challenge in large companies, as well as restrictions with regard to how data is stored and accessed, especially sensitive data that could be subject to various regulations and compliance requirements. A key to overcoming these challenges is to break down the silos by creating centralized data lakes and data discovery mechanisms, such as a searchable data catalog that contains metadata describing the various datasets in our data lake. To ensure that our data is stored and accessed securely, we need to implement robust encryption and permission-based access mechanisms. We will explore these topics in more detail in later chapters, and we will also perform some hands-on activities regarding detecting...