What is the modern challenge to integration?
Organizations have experienced challenges to integration for decades. Integration of applications across an enterprise has always been a complex task often made the more difficult because of early architectural design decisions. For example, point-to-point integration of two systems as described earlier in this chapter seemed like a good idea with lots of pros until the number of systems increased beyond two.
Truthfully though, increasing beyond two applications wasn’t the real problem. In many organizations, the real issue is either not prioritizing architecture as a discipline or employing “it’s just” architecture. As in, “it’s just one extra system, go ahead and have it connect to the database and read this table”. Because integrating three systems doesn’t seem like that much more than integrating two. And four systems don’t seem like that much more than three. Of course, the problem here is before long there are hundreds of critical systems in your organization, and you have ended up with the “big ball of mud” architecture shown earlier, unable to maneuver as quickly as the business wants to. IT is then left holding the bag trying to just keep the wheels on with no capacity to take on new projects or engage with the business to understand their ever-growing list of value-added ideas and innovation and desired business outcomes.
Fortunately, volumes have been written about integration across the enterprise because of all the challenges inherent to the effort. With all this history and with all these patterns, solutions, technologies, dissertations, architecture, methodologies, and platforms, we can now assert that the challenge of integration is resolved! Right?
Sadly, this is not the case. Let’s look at two primary reasons the industry has yet to find total enlightenment when it comes to integration.
Breaking the law is harder than you think
For many organizations, the law Melvin Conway introduced in 1967 has proven as difficult to overcome as Newton’s law of gravity from 1687. Conway’s law states the design of systems reflect the communication structure of the organization. In many of the organizations I have spoken with about integration, the chair-to-keyboard or swivel-chair integration approach to integration is the pattern used to connect applications because the applications exist in the same silos as the organization operating the application. The best they can hope for is an email, ideally with a spreadsheet attached so they can capture the parts they need in their own system. This is one of the same challenges observed by Hohpe and Woolf in their Enterprise Integration Patterns published in 2004. What they suggested was that the departments and the IT teams in an organization must change how they interact with each other if enterprise integration is to be successful.
Changing these policies takes an extraordinary effort. At one company, the CIO dictated a three-word strategy: best of breed. This was repeated all the way down through the ranks of the business and the IT organizations. The strategy described a policy which encouraged bringing in the very best system available for the corporate function needed. And if it couldn’t be found off the shelf, then build the best system. The strategy also established a policy to develop a central architecture team and an enterprise application integration (EAI) information bus platform capable of integrating these best of breed applications using asynchronous messaging with an enterprise canonical model.
The result was impressive. The mission critical applications in the organization were able to publish and subscribe to broadly agreed upon data structures. And the data from one system was made available to other systems in near real-time. However, this was not without significant costs and drawbacks. Maintenance and upgrade costs of the EAI platform, in addition to cost overruns of bespoke applications built when the marketplace couldn’t yield the called for best of breed application, eventually forced the organization to abandon the strategy. A new CIO called for a new strategy, fit for purpose, recommending the business look for COTS (commercial off-the-shelf) solutions bundling more functions together at the expense of any one of them being perhaps not quite the best you could build or buy.
While each CIO had their own reasons and their own strategy, the key to the success for each one was how well they were able to promote the new policy and gain buy in from all of the relevant stakeholders in the business and in IT.
Business innovation at the speed of technical debt
If the silos of business isn’t challenging enough, the speed at which businesses must innovate just to keep up may give an integration architect whiplash. Bundled with a load of technical debt however, can significantly slow the speed of innovation.
In his seminal work, The Innovators Dilemma, Clayton Christensen described how leading companies in businesses as diverse as computer manufacturers, hard drive manufacturers, and excavators were disrupted by innovative changes in technology. None of these great businesses were prepared for or able to move quickly when these new companies with new technology began to shift from idea to market leader. As Christensen describes it, the legacy firms are victims of their own success. And, in many of these cases, the new company or division also doesn’t have the technical debt held by the incumbent technology or company.
Today, innovation is being driven across most industries through digital transformation. Digital transformation of the companies culture. Digital transformation of the supply chain. Digital transformation of the sales and marketing. Digital transformation of the relationship with the customer. Now more than ever, having the right data at the right time from the right place is important, perhaps even critical, to the operations and long term success of a business.
At the same time many companies are asking their IT departments to do more with less. IT is asked to work faster and harder (often stated as “work hard play hard” to soften the blow). They’re asked to set up better automated deployment or change the project methodology to something leaner. But IT is very often just trying to keep the basic operations of all the systems under their remit running. So, the business finds a way which may involve spreadsheets, loading files, extracting files, or even standing up a simple web portal. Almost all of this technical debt which only adds to IT’s workload.
I observed a payroll process involving a spreadsheet of pay, extracted from a mainframe system and manually modified before attempting to load it into PeopleSoft. And the source for the paychecks going out? The spreadsheet itself, after loading it to PeopleSoft, was sent to the payroll company. Possibly after a few more “adjustments”. This makes digital transformation of payroll difficult if not impossible and is a daunting challenge for integration to tackle.
In some cases, IT has tried to get ahead of the curve by starting a cloud migration journey. Shifting assets and resources to a cloud provider has helped eliminate some of the overhead associated with maintaining all the servers and databases in the data center. The cloud can also help IT projects provision, start up, and shut down servers in minutes or hours. This sounds and works great until the request comes in to integrate this new system in the cloud with the systems still in the data center.
What can be done if complex, siloed organizations and siloed data and technical debt have stymied innovation in the IT department and the business leadership just showed up with a new idea to transform lagging sales and underperforming quarterly reports with digital innovation? Let’s look next at the capabilities and services in the MuleSoft platform which help address these issues and then why APIs are so important in the modern approach to integrating systems across the business.
