Architecture principles overview

The principles to be outlined next will give professionals a guiding light to make wise choices now that define future success. Research firms can be a good source for guidance, such as the following:

• Gartner {https://packt-debp.link/lXFfHo}
• Forrester {https://packt-debp.link/gEVSOD}
• Bain & Company {https://packt-debp.link/pAnKim}
• Frost & Sullivan {https://packt-debp.link/JrUuR0}
• International Data Corporation (IDC) {https://packt-debp.link/D4PrBA}
• 451 Research (now owned by S&P) {https://packt-debp.link/KF6rkc}
• Economist Intelligence Unit (EIU) {https://packt-debp.link/zy1quO}
• Experts Exchange {https://packt-debp.link/GvVItH}
• Omdia (was Ovum) {https://packt-debp.link/ZhkiPT}
• Nucleus Research {https://packt-debp.link/Wsg8SG}
• GigaOM {https://packt-debp.link/q1GLgK}
• Lexis Nexis Research {https://packt-debp.link/30GcOM}
• Olive Tree Insights {https://packt-debp.link/WX88AP}
• Aberdeen Strategy & Research {https://packt-debp.link/NQJFAu}
• Outsell {https://packt-debp.link/mmlLot}
• CEB (now owned by Gartner) {https://packt-debp.link/3mfY5h}

All the aforementioned produce valuable current insights for professionals trying to navigate the rough waters of our current technology journey. Sometimes the future is charted by riding the various hype cycles through their ups and downs. But is there a better way? Can the elusive future be made clear and plainly communicated with minimal cost?

Every development journey, including the development of the data factory’s data pipelines, begins with preparation and knowing what to pack. The choices are driven by principles. Following the herd will not get you to where you want to go; rather, it will get you only to where the herd is headed. These principles will become the constitution governing your architectural design choices. They will be anchors for conversations and settling disputes that will arise. Some of the most successful people, companies, and project efforts are those that have clear guiding directions for what they want and do not let babble (or babblers) get in the way of success.

In order to remain grounded as you make progress, you will want to develop the organization’s IT principles and remain securely anchored in them as you develop the architecture and the data engineering designs that are derived from this architecture. Be practical about the principles you develop from what will appear in this chapter. Be careful with the wording of the principles because each carries huge implications for the downstream development effort. Begin with a primer, and do not assume everyone is on the same level of knowledge as you formalize your principles. Your constituents, stakeholders, and engineers may have been drinking a vendor’s Kool-Aid for years and can’t talk about technology if it threatens their limited knowledge. Expect them to be defensive! The following sections were written to provide you with the foundation that will be used to level-set others with the knowledge necessary to gain acceptance for your solution. Many links are provided for you to deep dive into the subject matter. If you remain open to innovation and closed to compromises to quality, fragile design, short-sightedness, and waste, clear principles will arise that withstand the test of time.