The value of data architecture

Overall, a well-designed data architecture is essential for any organization that wants to get the most out of its data. By following the principles of simplicity and scalability, organizations can design a data architecture that meets their current and future needs. A well-designed data architecture can help organizations do the following:

• Drive implementation of strong data governance: By ensuring data is organized, stored, and processed efficiently, data architecture can reduce redundancies and inconsistencies.
• Improve data quality and consistency: By establishing standards and procedures for data collection, storage, and processing, data architecture can help to ensure that data is accurate, complete, and consistent across all systems.
• Increase data accessibility and usability: Data architecture can help to make data more accessible to users and easier to use for analysis. This can be done by developing data warehouses and data lakes, as well as by creating data models and dashboards.
• Enhance data security and compliance: Data architecture can help to protect data from unauthorized access and misuse. It can also help organizations to comply with data privacy and security regulations.
• Support data-driven decision-making: By making data more accessible and usable, data architecture can help organizations make better decisions based on data. This can lead to improved business performance and competitive advantage. A good data architecture facilitates data analytics, enabling businesses to derive valuable insights and make data-driven decisions.
• Ensure scalability: A well-designed data architecture should enable growth with the organization’s data needs and business needs as the company grows and expands. A tight connection with corporate strategy and data strategy should enable scalability to be considered appropriately.
• Enable security and compliance needs: Effective data architecture helps in maintaining data security and adhering to regulatory requirements.
• Drive cost efficiency: By optimizing data storage and processing, data architecture can lead to cost savings by simplifying storage costs, reducing the quantity of processing, and removing redundancy.

Optimizing for aggregation

Consider a very large organization with decades of historical data. Data comes into the organization in hundreds of source systems, connecting and moving point to point, with no “target-state” or “North Star” architecture. You can imagine that this leads to redundancies, data quality issues, difficulty in extracting data needed, when needed, and a huge inconsistency in information enterprise-wide.

In this particular instance, the company built a plan to work strategically over time to better optimize the flow of information. This required a strategic approach to how particular datasets were managed, how they moved throughout the organization, what common data was needed (see Chapter 10, Primary Data), and where it should be conformed.

Some key decisions that worked well in this build-out were the following:

• What were we optimizing for? Cost? Simplicity? Aggregation?
• Who was responsible for making difficult decisions?
• How fast would we rearchitect the company?

These decisions were used to define the North Star by which the data architecture was designed. When you are considering a data architecture for your organization, these questions can help you define what the “target state” might look like and how you will chart the path to implementation.

Why data architecture is often overlooked

Data architecture is often overlooked in organizations for a number of reasons, including a lack of understanding of the benefits. Many organizations do not fully understand the benefits of data architecture, or they may not even be aware of it. Many organizations are focused on short-term gains, such as meeting quarterly revenue targets. Data architecture is often seen as a long-term investment, and it can be difficult to justify the cost and effort upfront. It’s also an expensive endeavor to change, and thus, most companies lack the patience and focus to have the long-range plan realized. Furthermore, data architecture can be complex, and it can be difficult to know where to start. This can be especially challenging for smaller organizations with limited resources. There is also a shortage of skilled data architects, and it can be difficult to find and hire qualified candidates.

Together, these challenges often lead to data architecture being de-prioritized or simply a consideration but not a binding mechanism by which decisions are made. This often leads data and analytical professionals to additional hardship in serving their stakeholders due to underlying complexities within the company’s technical landscape.

However, as a data professional, you must advocate for data architectural rigor, because data architecture is an essential component of any organization that wants to get the most out of its data. A well-designed data architecture can help organizations to improve data quality, accessibility, security, and compliance. It can also support data-driven decision-making and lead to improved business performance.

Here are some tips for overcoming the challenges of data architecture:

• Start with a clear understanding of your business needs: What are your business goals? What data do you need to achieve those goals? Once you have a clear understanding of your business needs, you can start to develop a data architecture that meets those needs.
• Take a phased approach: Data architecture is a complex undertaking, and it is important to take a phased approach. Start by developing a high-level overview of your data architecture. Then, you can start to implement the different components of your data architecture in phases.
• Invest in training and development: If you do not have the in-house skills to develop and implement a data architecture, you may want to invest in training and development for your staff. You may also want to consider hiring a consultant to help you get started.

Helpful Hint

Remember—implementing a target state data architecture is a multi-year, sometimes 5-7-year journey. Because companies are not likely to implement an entirely new parallel ecosystem, the strategic transition is typically executed in many phases. The most successful implementations I have been a part of had multiple phases: the first being what could be done quickly, and the second, where tools and platforms could be consolidated or retired. Subsequent phases are when more material changes must be made and require additional investment. It’s critical to keep putting this in front of individuals making funding decisions so that they can be reminded where you are headed.

It can also be helpful to measure the success of data architecture as clearly as possible so that executive management can see the value in data architecture more clearly. In this next section, I will share some specific ways to measure data architecture value so that you can build your business case accordingly.

Measures of success

Measuring success in terms of business value and outcomes requires a combination of quantitative and qualitative metrics to ensure that the data architecture aligns with the organization’s strategic goals and delivers tangible benefits. Regular monitoring and adjustments are essential to maintain the effectiveness of the data architecture as business needs evolve. Ideally, these measures will be defined by the head of data architecture, aligned with the chief architect, and supported by the users of data, business data stewards, and technical data stewards. The enterprise data committee should approve both the to-be architecture and these measures of success.

Example metrics could include the following:

• Adoption: The number of users who have adopted the new data services or components of the prescribed data architecture.
• Data quality: The number of data quality issues resolved due to the new design.
• Requirements: The number of data requirements met with the new design.
• Costs: Cost of replication saved due to new design. Measure cost savings achieved through optimized data storage and processing.
• Accessibility: How easily users and applications can access the data they need by measuring the time from request to closure of ticket.
• Performance: Speed of data processing and/or efficiency of data retrieval.
• Security and compliance: Track compliance with security and/or privacy regulations. Ensure that sensitive data is identified within the architecture and that the architecture enables compliance (for example, do not put sensitive data in multiple places; prioritize simplifying and reducing the number of locations with sensitive data as a best practice).
• Scalability: Number of programs that leverage the new data architecture without major modifications.

Additionally, be sure to design measures of success that quantify business impact. Assess how data architecture contributes to achieving business goals and objectives. This could include increased revenue, cost reductions, improved customer satisfaction, or other relevant business outcomes. These will vary greatly based on your business. Pulling users and business and technical data stewards in to help define business impact metrics is a best practice.

While not a specific measure, you should continuously gather feedback from users and stakeholders and use it to adapt and improve the data architecture over time. You may want to establish some “program” metrics to demonstrate early progress, such as the definition of guiding principles, first draft to-be state, and quantity of feedback provided. This will show engagement in the process and is a measure of success in the definition stage.