Traditionally, enterprises have embraced data warehouses to store, process, and access large volumes of data. These warehouses are typically large RDBMS databases capable of storing a very-large-scale variety of datasets. As the data complexity, volume, and access patterns have increased, many enterprises have started adopting big data as a model to redesign their data organization and define the necessary policies around it.
This figure depicts how a typical data warehouse looks in an Enterprise:
As Enterprises have many different departments, organizations, and geographies, each one tends to own a warehouse of their own and presents a variety of challenges to the Enterprise as a whole. For example:
- Multiple sources and destinations of data
- Data duplication and redundancy
- Data access regulatory issues
- Non-standard data definitions across the Enterprise.
- Software and hardware scalability and reliability issues
- Data movement and auditing
- Integration between various warehouses
It is becoming very easy to build very-large-scale systems at less costs compared to what it was a few decades ago due to several advancements in technology, such as:
- Cost per terabyte
- Computation power per nanometer
- Gigabits of network bandwidth
- Cloud
With globalization, markets have gone global and the consumers are also global. This has increased the reach manifold. These advancements also pose several challenges to the Enterprises in terms of:
- Human capital management
- Warehouse management
- Logistics management
- Data privacy and security
- Sales and billing management
- Understanding demand and supply
In order to stay on top of the demands of the market, Enterprises have started collecting more and more metrics about themselves; thereby, there is an increase in the dimensions data is playing with in the current situation.
In this chapter, we will learn:
- Data architecture principles
- The importance of metadata
- Data governance
- Data security
- Data as a Service
- Data architecture evolution with Hadoop
