SOA building blocks

Let us now have a closer look at the SOA building blocks that enable us to realize the above-mentioned concepts:

• BPEL: This is for business process automation with service composition.

• Services: This is for achieving modular and flexible architecture. For service development, Web Services technology is usually used.

• Enterprise Service Bus (ESB): This provides a means for services and processes to communicate, and enables management and control over the communication. ESB is the backbone of SOA.

• Registries and repositories: These are central directories of services and useful for locating and reusing services, as well as SOA governance.

• Human task support: Business processes often involve human interaction. SOA supports human interactions in different ways, such as WS-HumanTask and BPEL4People. Human task support is related to Identity Management.

• Process monitoring or Business Activity Monitoring (BAM): This allows the monitoring of the execution of processes, such as total execution time, average execution time, execution time of certain activities, and so on. It also allows us to monitor the Key Performance Indicators (KPIs), which is particularly interesting for management, as it allows them to understand better how the business operations perform.

• Business Rules Management Systems (BRMS) or Rule Engine: This is a central place for managing business rules. With BRMS we can put business rules into a central location instead of hard coding them.

• Adapters: These provide easy access not only to external systems, such as ERP, CRM, SCM, but also DBMS systems.

A very important aspect of SOA is SOA governance. SOA is a complex architecture, which has to be governed in order to be consistent. SOA governance is a set of activities related to control over services and processes in SOA. Typical activities are related to managing service portfolios and lifecycles, ensuring service consistency, and monitoring service performance.

The full architecture of SOA is shown in the following figure:

The next figure shows the technology view of SOA and positions the above-mentioned concepts:

Let us now add the technologies into the previous figure to understand the connection between SOA concepts and the technologies that provide a means for their realization. Notice that the mere use of a specific technology does not guarantee that we are building an SOA-compliant architecture. For example, with Web Services we can develop business services (for example, a loan application), but we can also develop technology-focused services (updating the database, for example). So, it is essential that technologies are used according to the guidelines provided by SOA concepts.

For this figure, we can have two views. The bottom-up view of SOA sees different applications exposing their functionalities through business services. This enables access to functionalities (services) of different existing and newly developed applications in a standard way. Access to services is important because a typical enterprise has a large number of applications that have to be integrated.

Developing business services, either through reuse of existing applications or by new development, is not sufficient. We also need to compose services into business processes this is the second, the top-down, or process-oriented approach to SOA. We would obviously prefer a relatively simple and straightforward way to compose and modify business processes. This is where BPEL becomes important.

BPEL for process automation

Services in SOA are composed into aggregate services. We compose services until the aggregate services provide support for the whole business processes. Business processes are thus defined as a collection of activities through which services are invoked. For the outside world (that is, for the clients) a business process looks like any other service. In real-world scenarios we will usually create two kinds of business processes those that will contain services from within the enterprise only, and those that will consume services provided by several companies. With service composition, we can use services provided by business partners in our processes, and business partners can use our services in their processes.

For example, a business process for booking business travel will invoke several services. In an oversimplified scenario, the business process will require us to specify the employee name, destination, dates, and other travel details. Then the process will invoke a service to check the employee's status. Based on the employee status, it will select the appropriate travel class. Then it will invoke the services of several airline companies (such as American Airlines, Delta Airlines, and so on) to check the airfare price and buy the one with the lowest price. The structure of services composed in the business process is shown in the following figure. In Chapter 2, we will discuss this example in detail and show how to define this process using BPEL.

From the perspective of our business process, we do not care whether the service for checking the employee status accesses a legacy system, a database directly, or retrieves the status in any other way. We also do not care whether the services of airline companies are composed of other, lower-level services. From the perspective of the client (for our business process), the client sees the process as any other service and does not care whether the process is implemented through composition of other services, or some other way. This stimulates reuse and further composition. Real-world business processes will usually be much more complicated than our example. Usually they will contain several services and invoke their operations either in sequence or in parallel. They will also contain flow logic, handle faults, take care of transactions and message correlation, and so on.

The composition of services into business processes requires the definition of collaboration activities and data-exchange messages between involved Web Services. WSDL provides the basic technical description and specifications for messages that are exchanged. However, the description provided by WSDL does not go beyond simple interactions between the client (sender) and the web service (receiver). These interactions may be stateless, synchronous, or asynchronous. These relations are inadequate to express, model, and describe the complex compositions of multiple Web Services in business activities, which usually consist of several messages exchanged in a well-defined order. In such complex compositions, synchronous and asynchronous messages can be combined, and interactions are usually long running, often involving state information that has to be preserved. An important aspect is also the ability to describe how to handle faults and other exceptional situations. Given the limitations of WSDL, we need a mechanism to describe the composition of Web Services into more complex processes.

In addition to these facts, the composition of business processes has other specific requirements, such as support for many process instances, long-running processes, compensation, and so on. All this makes the use of dedicated solutions reasonable. This is why over the years several proprietary BPM (Business Process Management) products have been developed, such as Dralasoft Workflow and TIBCO Business Process Management. The disadvantage of using proprietary BPMs is that these are traditionally niche products, sold from a top-down perspective to large business users. Such products are usually expensive and bound to a certain provider. This is why we need BPEL.

Note

BPEL is equally important for SOA as SQL is for databases.

Web Services

Web Services are the latest distributed technology and, as we will see, the most suitable technology for the realization of SOA. They have become the commonly used technology for interoperability and the integration of applications and information systems. Web Services provide the technological foundation for achieving interoperability between applications using different software platforms, operating systems, and programming languages. They are built on XML. While XML is the de facto standard for data-level integration, Web Services are becoming the de facto standard for service-level integrations between and within enterprises.

From the technological perspective, Web Services are a distributed architecture. The distributed computing paradigm started with DCE (Distributed Computing Environment), RPC (Remote Procedure Call), and messaging systems, also called message-oriented middleware (products such as MQSeries, MSMQ, and so on). Then distributed objects and ORBs (Object Request Brokers) such as CORBA (Common Object Request Broker Architecture), DCOM (Distributed Component Object Model), and RMI (Remote Method Invocation) emerged. Based on these, component models, such as EJB (Enterprise Java Beans), COM+ (Component Object Model), .NET Enterprise Services, and CCM (CORBA Component Model) have been developed. RPC, ORBs, and component models share a similar communication model, which is based on a synchronous operation invocation. Messaging systems are based on an asynchronous communication model.

How Web Services differ from their predecessors

Web Services are similar to their predecessors, but also differ from them in several aspects. Web Services are the first distributed technology to be supported by all major software vendors. Therefore, they are the first technology that fulfills the promise of universal interoperability between applications running on disparate platforms. The fundamental specifications that Web Services are based on are SOAP (Simple Object Access Protocol), WSDL (Web Services Description Language), and UDDI (Universal Description, Discovery, and Integration). SOAP, WSDL, and UDDI are XML based, making Web Services protocol messages and descriptions human readable.

From the architectural perspective, Web Services introduced several important changes compared to earlier distributed architectures:

• Web Services support loose coupling through operations that exchange data only. This differs from component and distributed object models, where behavior can also be exchanged.

• Operations in Web Services are based on the exchange of XML-formatted payloads. They are a collection of input, output, and fault messages. The combination of messages defines the type of operation (one-way, request/response, solicit response, or notification). This differs from previous distributed technologies. For more information, please refer to the WSDL and XML Schema specifications (XML spec: http://www.w3.org/TR/ REC-xml/ and WSDL spec: http://www.w3.org/TR/wsdl).

• Web Services provide support for asynchronous as well as synchronous interactions.

• Web Services introduce the notion of endpoints and intermediaries. This allows new approaches to message processing.

• Web Services are stateless. They do not follow the object paradigm.

• Web Services utilize standard Internet protocols such as HTTP (Hyper Text Transfer Protocol), SMTP (Simple Mail Transfer Protocol), FTP (File Transfer Protocol), and MIME (Multipurpose Internet Mail Extensions). So, connectivity through standard Internet connections, even those secured with firewalls, is less problematic.

Web Services technology stack

In addition to several advantages, Web Services also have a couple of disadvantages. One of them is performance, which is not as good as that of distributed architectures that use binary protocols for communication. The other is that plain Web Services do not offer infrastructure and Quality of Service (QoS) features, such as security, transactions, and others, which have been provided by component models for several years. Web Services fill this important gap by introducing additional specifications:

• WS-Security: Addresses authentication and message-level security, and enables secure communication with Web Services.

• WS-Coordination: Defines a coordination framework for Web Services and is the foundation for WS-AtomicTransaction and WS-BusinessActivity.

• Transactions specifications (WS-AtomicTransaction and WS-BusinessActivity): Specify support for distributed transactions with Web Services. AtomicTransaction specifies short duration, ACID transactions, and BusinessActivity specifies longer-running business transactions, also called compensating transactions.

• WS-ReliableMessaging: Provides support for reliable communication and message delivery between Web Services over various transport protocols.

• WS-Addressing: Specifies message coordination and routing.

• WS-Inspection: Provides support for the dynamic introspection of web service descriptions.

• WS-Policy: Specifies how policies are declared and exchanged between collaborating Web Services.

• WS-Eventing: Defines an event model for asynchronous notification of interested parties for Web Services.

Because of their flexibility, interoperability, and other features, Web Services are regarded as the most appropriate technology for exposing the functionalities of applications as services and are therefore the most appropriate technology for realization of SOA. Because of their wide support by all major software vendors, Web Services provide the possibility to use the same technology to expose services implemented in a variety of different applications ranging from mainframe-based legacy applications to the modern multitier applications.

Enterprise Service Bus

While Web Services are an appropriate technology for SOA, some other aspects need to be considered, such as:

• In most enterprises, Web Services are not the only middleware solution used. Usually enterprises already have one or more middleware products, such as messaging systems and ORBs. Enterprises cannot afford to replace them overnight with Web Services. Therefore, there is a need to integrate different middleware products, and provide interoperability with Web Services.

• In order to provide connectivity between services, the use of SOAP in complex environments is not adequate. In such environments, we need ways to connect, mediate, manage, and control the services and particularly the communication between them.

• SOAP over HTTP might not be robust enough for heavy enterprise use. Enterprise information systems require dependable, robust, and secure service infrastructure.

The ESB is the software infrastructure, acting as an intermediary layer of middleware that addresses the above-mentioned requirements. An ESB adds flexibility to the communication between services, and simplifies the integration and reuse of services. An ESB makes it possible to connect services implemented in different technologies (such as EJBs, messaging systems, CORBA components, and legacy applications) in an easy way. An ESB can act as a mediator between different, often incompatible protocols and middleware products.

The ESB provides a robust, dependable, secure, and scalable communication infrastructure between services. It also provides control over the communication and control over the use of services, including:

• Message interception capabilities: This allows us to intercept requests to services and responses from services, and apply additional processing to them. In this manner, the ESB acts as an intermediary.

• Routing capabilities: This allows us to route the messages to different services based on their content, origin, or other attributes.

• Transformation capabilities: These allow us to transform messages before they are delivered to services. For XML-formatted messages, such transformations are usually done using XSLT (Extensible Stylesheet Language Transformations) or XQuery engines.

• Control over the deployment, usage, and maintenance of services: This allows logging, profiling, load balancing, performance tuning, charging for use of services, distributed deployment, on-the-fly reconfiguration, and so on.

• Other important management features: These include the definition of correlation between messages, definition of reliable communication paths, definition of security constraints related to messages and services, and so on.

ESB features

The ESB enables the communication and management of services, along with providing answers related to the usage of services in complex enterprise information systems. In such environments, support for the centralized, declarative, and well-coordinated management of services and their communication is required. Because of existing middleware, the integration of different middleware products and interoperability with other services is required.

The ESB is important because it represents the communication backbone for services. Using ESB for communication, we can get rid of point-to-point connections between services and processes. ESB also simplifies the transition between development, test, and production environments.

The most important features of ESB are service routing, transformation and enhancement, protocol transformation, service mapping, security, and Quality of Service.

• Message routing enables us to route messages to a specific service provider in a declarative manner based on the message content, user type, channel, or other rules.

• Message transformation enables us to transform the input and output messages. Usually this is related to XML transformation using XSLT.

• Message enhancement enables us to add data to the message or remove data, so that they conform to the requirements of the service provider and service consumer.

• Protocol transformation is the ability to automatically transform the protocol based on the service provider and service consumer preferences. For example, a service consumer might use SOAP, while the service uses JMS. Protocol transformation can also optimize performance and switch to an optimized protocol for collocated services.

• Service mapping enables us to map a service to a specific service implementation. This is usually an extension of WSDL bindings.

• Security enables us to secure services and the transportation layer used for the exchange of messages. For securing services, authentication and authorization are important; for securing the exchange of messages, encryption is usually applied.

• Quality of Service allows us to define specific parameters of service communication, such as reliability, bandwidth, availability, and so on. Quality of Service assurance is the baseline for the definition of Service Level Agreements (SLAs).

Currently, there are several products in the market that claim to provide ESB functionality. A good ESB should provide at least Quality of Service support at enterprise level, including reliability, fault-tolerance, and security. If provided by an ESB, services can depend on these features and do not need to implement them themselves. The ESB should also allow the configuration of any combination of these Quality of Service features and provide flexibility.

An ESB should provide support for a variety of technologies on which services are implemented. In addition to Web Services, an ESB should provide connectors for a broad range of technologies, such as Java EE and .NET components, messaging middleware, legacy applications, and TP monitors. The ESB needs to provide flexibility to bind any combination of services without technological constraints. It should also support a combination of different interaction models, such as queuing, routing, and so on, without changing the services or requiring code writing.

An ESB should make services broadly available. This means that it should be easy to find, connect, and use a service irrespective of the technology it is implemented in. With a broad availability of services, an ESB can increase reuse and can make the composition of services easier. Finally, an ESB should provide management capabilities, such as message routing, interaction, and transformation, which we have already described.

An ESB that provides these features becomes an essential part of the SOA. It provides several benefits, including increased flexibility, reduced deployment, development, and maintenance costs, along with increased reliability and manageability.

The following figure shows the relation between BPEL, ESB, and services:

Registry and repository

We have already seen that SOA consists of services and processes. The more applications that we implement following SOA concepts, the more services we will have. Managing services becomes a challenging task. This includes aspects such as:

• How many services do we have?

• Does a service with a specific functionality already exist?

• Where is the service deployed?

• Which versions of the service exist?

• Who are the consumers (clients) of a specific service?

• What is the interface definition (WSDL) of a service?

Registries and repositories help to answer these and similar questions. They have become an essential part of each SOA architecture. Registries and repositories are used to register services in a central location. Once registered, we can search and locate appropriate services. The more metadata about a service we include, the better search capabilities a registry and repository can provide.

A rule of thumb is that once we have more than 50 services we will desperately start needing the registry and repository. However, sometimes it might be wise to introduce it from the beginning because once developers get used to a certain development process, it will be very difficult to change their behavior.

In addition to the above-mentioned questions, registries and repositories play an important role in service reuse. Somewhat simplifying things, reuse on a service level means that we compose executable processes (in BPEL) and reuse as many services as possible instead of developing them from scratch.

Registries and repositories also play an important role in the deployment process. In professional development it is usual to have three environments development, test, and production. When an application is ready for production use, this requires that it is deployed into the production environment. The controlled deployment of processes requires that a process starts to use a production version of services instead of development or test versions. If links to services are hard coded into the BPEL processes, such migrations can be very painful, as it might (in a worst-case scenario) require the manual change of URL addresses. When deploying a service we might want to know which processes use this service because we might want to retest those processes.

The understanding of the role of registries and repositories in SOA has changed considerably over the years. A few years ago, we believed that a relatively simple registry, UDDI, would cover all the needs. Today, we have identified that a registry alone is not powerful enough for SOA because in many cases it makes sense to store service metadata as well (WSDL interfaces, XSD schemas, and so on).Therefore, today we talk about registries and repositories.

A powerful registry and repository should have the following features:

• Classification capabilities to categorize and classify services and processes based on one or more classification schemas. This simplifies queries and enables easier locations for the most appropriate services for reuse.

• Governance functions that should enable the definition of proprietary service/process lifecycles, together with the conditions to go from one stage of the lifecycle to another. Stage transitions can trigger automatic actions, such as validators.

• Access control that defines who can do which operations on the registry and repository, and on the registered services/processes. Such access control could be based on XACML (eXtensible Access Control Markup Language).

• User, programming, and administration interfaces.

Registries and repositories also play an important role in SOA governance.

Human task support and identity management

Business processes often involve human interaction. SOA, therefore, has to provide support for human tasks. Human tasks are sometimes called user interactions. Human tasks in business processes can be simple, such as approving certain tasks or decisions, or complex, such as delegation, renewal, escalation, nomination, or chained execution. Task approval is the simplest, and probably the most common, human task. If the situation is more complex, a business process might require several users to make approvals, either in sequence or in parallel. In sequential scenarios, the next user often wants to see the decision made by the previous user. Sometimes, (particularly in parallel human tasks) users are not allowed to see the decisions taken by other users. This improves the decision potential. Sometimes one user doesn't even know which other users are involved, or whether any other users are involved at all.

A common scenario for involving more than one user is escalation. Escalation is typically used in situations where an activity doesn't fulfill a time constraint. In such a case, a notification is sent to one or more users. Escalations can be chained, going first to the first-line employees and advancing to senior staff if the activity is not fulfilled.

Sometimes it's difficult or impossible to define in advance which user should perform an interaction. In this case, a supervisor might manually nominate the task to other employees; the nomination can also be made by a group of users or by a decision-support system.

In other scenarios, a business process may require a single user to perform several steps that can be defined in advance or during the execution of the process instance. Even more-complex processes might require that one workflow is continued with another workflow.

Human tasks can include data entries or process management issues, such as process initiation, suspension, and exception management. This is particularly true for long-running business processes, where, for example, user exception handling can prevent costly process termination and related compensation for those activities that have already been successfully completed.

As a best practice for human tasks, it's usually not wise to associate human tasks directly to specific users. It's better to connect tasks to roles and then associate those roles with individual users. This gives business processes greater flexibility, allowing any user with a certain role to interact with the process and enabling changes to users and roles to be made dynamically. To be able to do this, we need to have an identity management system, where users, roles, and groups are managed. This can be a simple LDAP or a more sophisticated system.

To interleave human tasks with BPEL processes we can use a workflow service, which interacts with BPEL processes using standard WSDL interfaces, as any other service. This way, the BPEL process can assign user tasks and wait for responses by invoking the workflow service using the same syntax as for any other service. The BPEL process can also perform more complex operations such as updating, completing, renewing, routing, and escalating tasks.

To standardize the explicit inclusion of human tasks in BPEL processes the BPEL4People (WS-BPEL Extension for People) specification has been proposed. BPEL4People introduces people activities and people links into BPEL. People activity is a new BPEL activity used to define user interactions, in other words, tasks that a user has to perform. For each people activity, the BPEL server must create work items and distribute them to users who are eligible to execute them. To specify human tasks the WS-Human Task specification has been proposed.

Process Monitoring or Business Activity Monitoring

One of the key elements for process control is Process Monitoring or Business Activity Monitoring (BAM). The key objective of BAM is to provide a complete insight into business process execution. BAM allows for the monitoring of KPIs, such as total execution time, average execution time, the execution time of certain activities, and so on. This allows us to better understand how the business processes perform.

Note

Business Activity Monitoring is the real-time observation of key performance indicators.

The most important component of BAM is time. Time is crucial because BAM shows the actual near real-time information on process execution. This way, a company can react quickly and efficiently to changes reflected through process execution.

BAM solutions can provide other useful information. They can show us how many process instances are active at a specific time, how long on average it takes to complete a process, which users (employees) have started how many process instances, and so on.

Note that BAM is not related to automatic activities (those implemented by services) only. It can be used with human activities as well. In such a case, we can use BAM to observe the productivity of employees.

BAM is not only a system that displays interesting information about processes, but also consolidates data gathered from different, often independent sources. Connecting these data with past data enables BAM to identify critical situations in process execution or even automatically or semi-automatically solve some frequent critical problems. The ultimate goal of each BAM user is to optimize the process execution, to improve the process efficiency, and to sense important events and react.

The BAM user interface (dashboard) should be simple and present information and data in an easy and understandable way. It should hide all the complexity behind the scenes. Usually a typical BAM user interface uses graphical elements, graphs, and colors to present the data. The next screenshot shows an example of a BAM user interface:

In the previous screenshot, we can see the BAM dashboard showing different important information for decision makers. In addition to the dashboard, another important part of the BAM is the decision-support module. A module such as this can use decision methods, business intelligence, or simulations for support and can help decision makers take the right decision at the right time, which can improve business efficiency.

Business Rules Management Systems (BRMS) or Rule Engine

Business rules are part of almost every business application. Experiences have shown that business rules change, sometimes very often. Today, business rules are usually coded into different applications and are tightly coupled with the implementation of an application system. Changing business rules is therefore often very difficult and requires modifications to applications. Each modification requires that the developer change the code. After that, testing and deployment has to be done.

The same business rules are often implemented in several applications. Therefore, if a rule changes, the major challenge becomes to identify which applications this business rule is coded into and to modify each application where such a rule has been used.

Business rules are also very common in business processes. Business Rules Management Systems (BRMS) or Rule Engines are meant to provide a central location for storing, executing, and managing business rules. Instead of hard-coding business rules into the executable code (whether BPEL, Java, C#, and so on), we place business rules into the BRMS, where:

• Business rules can be reused from different processes, services, and applications

• User-friendly interfaces exist, which enable us to change and modify business rules

Adapters

Adapters in SOA are meant to simplify the integration with external systems, such as ERP, CRM, SCM, and others. Without adapters we would need to manually expose functionality out of such systems, for example by developing corresponding Web Services. As such systems usually have rich functionalities, it would require a considerable amount of time and effort to develop the integration interfaces and services. Additionally, when such a system is upgraded, we would need to modify the integration services.

Adapters automate this procedure and provide tools to integrate with such systems in an easy way. Examples of adapters are adapters for SAP, Oracle, PeopleSoft, Salesforce, or some other popular system. Adapters are also useful for accessing database data directly. Such adapters automate the development of data services. Examples include adapters for Oracle, DB2, SQL Server, and so on.

Adapters differ in the level of sophistication. The simplest adapters are just API wrappers that expose the interfaces of the external systems as a service (usually as a web service). More sophisticated adapters can have smart fault handling mechanisms, can capture events on both sides, allow synchronous and asynchronous interactions, take care of load balancing, performance, scalability, security, and so on.

When deploying an SOA platform, it makes sense to check whether it provides adapters for the systems that we have deployed in our information system. With adapters, accessing the functionality of such systems will most likely be much simpler than without.

Service Component Architecture

So far we have seen that SOA applications are composite applications that consist of several components, such as services, BPEL processes, ESB mediation, rules, adapters, and so on. All these components have to work together and support one or more composite applications.

Service Component Architecture (SCA) defines a programming model for composite SOA applications. SCA is based on the idea of service composition (orchestration). SCA provides a model for the composition of services and for the creation of service components, including the reuse of existing applications within SCA composites.

The basic building blocks of SCA are components. Every SCA application is built from one or more components. Components can be implemented in a programming language, such as Java, C++, C, or they can be implemented as BPEL processes. SCA provides a generalized definition of a component.

Components offer their business functions for use by other components as services. Implementations may depend on services provided by other components. Such dependencies are called references. Implementations can have properties through which the business function can be influenced. The component configures the implementation by providing values for the properties and by wiring the references to services provided by other components. An SCA component is represented as follows:

Service components are assembled into applications, called composites. Composites can contain components, services, references, properties, and wiring. Composites themselves can also be used as components and can provide services. They can depend on references and expose properties.

Composites can be used within other composites. This allows a hierarchical construction of composite business applications, where high-level services are implemented by sets of lower-level services. The following example shows an SCA composite:

Composites are deployed within an SCA Domain. An SCA Domain represents an area of business functionality with the related services. The SCA Domain diagram is shown in the next figure:

We will use the SCA and composites in Chapter 4, when we start to develop composite applications.

SOA governance

SOA governance is a set of related activities for exercising control over services and processes in SOA. SOA is a distributed architecture where applications consist of services and processes. The objective of SOA governance is assurance that all services and processes within an information system will adhere to the overall architecture. SOA governance has to address many different aspects, among them:

• Services and processes must follow specific guidelines, best practices, and patterns

• Services should not be duplicated

• Services are reused and reusable

• The portfolio of services is managed, including the development of new services and modifications of existing services

• Versioning of services and processes is defined

• Service lifecycle is defined

• Deployment practices are defined

• Consistency of services is monitored

• Quality of Services is ensured

SOA governance is an essential element of a successful SOA implementation. Without governance, SOA will most likely fail to deliver its value to the stakeholders. To learn more about SOA governance, have a look at SOA Governance by Todd Biske, Packt Publishing.