Getting Started with BizTalk Services

Chapter 1. Hello BizTalk Services

As companies build and extend their IT information assets, there is a need to integrate applications to exchange data. While a point-to-point application integration using custom code is possible, it does not address large-scale application integrations without duplication of data and the complexity of managing such an integration. BizTalk is a de facto choice for message-based integration for on-premises systems and for services on Azure for both applications and businesses.

In this book, we're going to introduce you to BizTalk Services, Microsoft's new integration middleware hosted on Windows Azure. This book assumes that you already understand the need for integration and the many benefits of using specialist integration software as opposed to building custom point-to-point integration solutions. We will look at why cloud-hosted integration services are so compelling and the scenarios in which these services make most sense. Prior knowledge of BizTalk Server is not expected, though it will help you to quickly understand certain topics within the book. We will assume you are a professional developer, an architect, or a solution designer familiar with the challenges of integrating heterogeneous systems and applications. The book also assumes familiarity of the Microsoft developer toolset, specifically .NET, Visual Studio, and SQL.

In this chapter, we will start to answer some fundamental questions around what BizTalk Services is and how it can help you to build integration solutions. Specifically, the following topics will be discussed:

The business and technical drivers for BizTalk Services on Azure
Concepts and architecture of BizTalk Services
Using BizTalk Services to realize a simple purchase order scenario

Background

First though, some background is necessary. At the Professional Developers Conference (PDC) 2008, Microsoft unveiled Windows Azure—a new operating system designed for the cloud. Over the subsequent years, Windows Azure has become Microsoft's de facto cloud platform covering services, media, websites, mobile applications, and more. While BizTalk Server has been an established product for over ten years (and eight releases), cloud adoption has been driving connected systems across different services and Line-of-Business applications. To meet the growing demand for cloud-based integration, the Microsoft BizTalk team released the first version of BizTalk Services, named Service Bus EAI and EDI Labs Community Technology Preview (CTP) on December 17, 2011. The goal was for customers to be able to sign up in a shared environment and set up simple XML/EDI flows without worrying about installation and maintenance. The capabilities were rich enough to enable simple point-to-point integration scenarios; on April 9th of the following year, the CTP was refreshed, incorporating feedback from customers by providing additional capabilities.

The CTP environment was hosted on a publicly shared service, and therefore had restrictions on running users' custom code. Integration is rarely straightforward, and the ability for developers to write custom code and deploy it as part of their integration solutions was a key requirement. Customers had also needed guarantees around performance and Service Level Agreements (SLAs). Hosting user assemblies as part of the cloud services became a reality by switching to per tenant deployment. Thus, the cloud offering known as BizTalk Services was born on June 3, 2013. Like many Azure services, BizTalk Services is expected to be updated at a regular cadence. The latest update as of this writing was on February 20, 2014. This technology is opening up new integration possibilities; with Microsoft's on-going investments, it will be on par with the capabilities of its on-premises cousin, BizTalk Server, in the near future.

Business drivers

There are many tangible benefits of building solutions on Azure today. A few of these include the ability to scale up/down the platform based on predictable or dynamic shifts in application load and throughput requirements without worrying about the hardware procurement time and setup, the ability to pay as you go with expense incurred as an Operational Expenditure (OpEx) instead of a Capital Expenditure (CapEx), and the increase in reach for certain integration scenarios.

Specific to integration on Azure, there are four factors that drive adoption:

Focus on business operations, not IT: There are business benefits in terms of reduced cost by leveraging platforms running under economy of scale, making it cheaper for customers to obtain them with higher quality of services. The need of the hour is to simplify IT deployment and management and focus on business services instead of configuring software or hardware.
Simplicity of managing externally facing services: Enterprises typically offer services across geographic and organizational boundaries. Many of these services require making changes to the corporate firewall to allow or deny the applications. This process is an IT nightmare for many organizations. With Azure integration services such as B2B, which inherently require external communication, all this could be moved to the cloud and the necessary access and control policies could then be centralized for all internal services. This also enables self-service configuration changes to the application, thereby dramatically improving responsiveness to business changes.
Greenfield cloud applications: The proliferation of mobile devices, such as smartphones and tablets as well as Point-of-Sale (POS) systems, has given rise to services that are inherently cloud based. Think of a POS that transmits daily transaction logs to its backend Line-of-Business systems or an RFID service that transmits information about each item in a shopping cart purchased in a retail store to an inventory application. As new services are being rolled out, organizations want to be able to develop and deploy these services with shorter time-to-market using Azure.
Another factor to consider for the cloud is to leverage existing on-premises investments. Businesses have invested in a variety of on-premises systems, including traditional ERP as well as legacy mainframes and other bespoke systems that literally run the business.

Concepts

The following figure illustrates a basic integration flow from an FTP source to a LOB destination. BizTalk Services, represented by the middle box, is a sequence of processing steps.

Role of BizTalk Services

BizTalk Services introduces several key concepts to facilitate building integration solutions on Azure:

Bridge: A bridge is a unit of processing in BizTalk Services that can address impedance mismatch. It contains three units: one or more source locations (for example, FTP) to read messages from, a pipeline to process the message, and one or more destinations (for example, Queue) to write the processed messages. The pipeline is divided into distinct processing units called stages, each with its own function (for example, a stage in a pipeline can validate a message against a schema). A series of stages represents the bridge pattern or bridge template. Out of the-box, BizTalk Services v1 ships with three templates: XML, EDI, and AS2.
Adapter: An adapter is the transport medium that can send messages (to a destination) or receive messages (from a source) and pass them to the pipeline in a bridge; for example, Line-of-Business adapters such as SAP and Oracle EBS or transport adapters such as FTP and SFTP.
Transform: A transform converts a message from one format into another, aiding structural conversion. Transforms contain operations that can perform commonly used transformations like string operations, loop constructs, list operations, and arithmetic and logical expressions.
Application protocol: A protocol defines the message format and processing semantics such as the requirement to send and correlate acknowledgements of messages.
Route: A route defines the destination endpoint where the message will be sent based on the specified criteria. The route criteria are evaluated based on SQL-92 expression syntax.
Batching: The aggregation of messages based on selection criteria is termed batching. The release (sending) of a batch is governed by size, count, or time, or a combination of these parameters.
Promoted properties: Promoted properties are name-value pairs, where the name is user-defined and the value is derived from the message header, message body, or from the context within the bridge. Promoted properties are commonly used in batching and routing to specify their criteria.
Artifact: An artifact is anything that aids in the processing of the message in the bridge. XML schemas, maps, custom assemblies, and certificates are the artifacts used in XML and EDI bridges. In BizTalk Services, each artifact is stored in the artifact store and is addressable by a unique URL.

Life cycle and architecture

Unlike most other Azure services deployments, BizTalk Services provisions dedicated resources for storage and compute instances that are isolated across tenants. This means that no two deployments have anything in common between them. The advantage is that you can write any custom code and be assured that you cannot impact the performance or availability of other deployments. This dedicated deployment also offers isolation of data at the storage level, thus increasing the privacy of data and SLA of the service.

Broadly categorizing, there are three steps to go through before you can have an active usable deployment. The first is to provision the service using standard Azure tools, including the Azure portal to create the service; the second is to deploy the necessary artifacts and configuration outlined in the BizTalk Services concepts section using Visual Studio and the BizTalk Services portal; and the third is sending or receiving the messages.

The architecture of BizTalk Services contains three key components:

Provisioning services: This is a set of Microsoft services that manage the lifecycle of a BizTalk Services deployment as well as monitor its health. It also includes components to bill the end user based on usage of BizTalk Services. The management interface to the service is exposed via the Red Dog Front End (RDFE) public API. The Azure Management Portal or PowerShell scripts from the user go via the RDFE API. Using the service, you can scale up/down your deployment as well as back up and restore deployment across datacenters.
Note
Red Dog was the original codename for Azure, with the "FE" being the publicly accessible frontend that users directly interact with either via the Azure portal or service management APIs.
Per tenant BizTalk Services: This is the per tenant deployment that is created in the user's Azure subscription. A BizTalk Services deployment is identified by the deployment name and is accessible using the URL secured by the Access Control Service (ACS). All artifacts such as bridges and schemas are added into the deployment with a URL which is a sub-path of the deployment URL. For example, a bridge is added under <deployment URL>/default/<bridgeName>. Here default is the namespace name where the artifacts are grouped into.
Per tenant dependencies: These dependencies are the Azure services required for tracking, troubleshooting, and security. For example, BizTalk Services provides a tracking store, which is an Azure SQL database where the processing status of messages, along with related properties, are stored as the messages pass through a bridge. The information from the tracking store is shown in the BizTalk Services portal tracking view. Archiving and monitoring is stored in Azure storage blobs and tables. Archived messages are stored in blob containers based on the date of the archive. The storage also contains the Azure table WADLogsTable, where tracing information for a bridge can be obtained. Finally, Access Control Service regulates access to all endpoints in the deployment. During deployment creation, the provisioning service uses the Management Service credentials to programmatically access ACS to create a relying party for the BizTalk Services deployment, add rule groups for Send, Listen, and Manage claims, and create the service identity with the necessary passwords for directly talking to ACS for deployment of the artifacts. The interaction between these components is illustrated in the following figure:
Block diagram of BizTalk Services shared and per tenant services

Personas and tools

BizTalk Services provides different user experiences for different personas to facilitate an optimized, task-based approach. The principal personas are:

Persona	Description	Primary Tools
Developer	Someone who creates integration solutions and artifacts, such as transforms and schemas	Visual Studio
IT Pro	Someone who manages the environment, including tasks such as deployment, setup, and configuration	Azure Portal and dashboard
Partner Administrator	Someone who sets up and manages trading partners	Trading partner/BizTalk portal

Developer

A developer will typically focus on creating solutions using Visual Studio. BizTalk Services VS 2012 project templates are provided to enable rapid creation of both EAI and EDI solutions. These provide a graphical work surface on which to create and configure bridges to facilitate communication between an enterprise and its trading partners. Additionally, sophisticated tools are provided, including a graphical mapping interface and schema editors.

IT Pro

The Windows Azure Platform Management Portal provides access to create the BizTalk Services deployment and management tasks as well as at-a-glance status information providing details on the overall health of all deployments and accounts. In addition to services deployment, the Windows Azure Platform Management Portal provides an interface for provisioning Azure SQL databases, mobile services, Service Bus entities, and so on.

Partner Administrator

The Partner Administrator persona uses the BizTalk Services portal for a number of management functions such as the creation and administration of trading partners, configuration of agreements including required transformations, routing and acknowledgements, tracking of messages, and exception processing.

The BizTalk Services portal enables the creation of trading partners and agreements between them. This enables the setting up and management of the protocols used to exchange data (for example, X12 and AS2) and the message formats to use together with transformation and routing capabilities. In this way, trading partners can be onboarded and configured quickly and easily by non-IT personnel without the use of developer tools such as Visual Studio, all through the web-based portal.

In addition, the management portal also provides the ability to set and view tracking data on message flows, including both contextual details (sender, message type, and so on) as well as the message bodies themselves. The ability to archive and export message data is also provided as part of the service.

Additionally, RESTful APIs are implemented to provide full fidelity with the portal, enabling activities to be scripted and deployment to be automated. Additionally, integration with customer systems and tools such as SharePoint for tracking data, visualization, or on-premises storage is also possible using this API.

Provisioning BizTalk Services

BizTalk Services deployment can be created using the Windows Azure Management Portal or using PowerShell. We will use the former in this example.

Certificates and ACS

Certificates are required for communication using SSL, and Access Control Service is used to secure the endpoints of the BizTalk Services deployment. First, you need to know whether you need a custom domain for the BizTalk Services deployment. In the case of test or developer deployments, the answer is mostly no. A BizTalk Services deployment will autogenerate a self-signed certificate with an expiry of close to 5 years. The ACS required for deployment will also be autocreated. Certificate and Access Control Service details are required for sending messages to bridges and agreements and can be retrieved from the Dashboard page post deployment.

Storage requirements

You need to create an Azure SQL database for tracking data. It is recommended to use the Business edition with the appropriate size; for test purposes, you can start with the 1 GB Web edition. You also need to pass the storage account credentials to archive message data. It is recommended that you create a new Azure SQL database and Storage account for use with BizTalk Services only.

The BizTalk Services create wizard

Now that we have the security and storage details figured out, let us create a BizTalk Services deployment from the Azure Management Portal:

From the Management portal, navigate to New | App Services | BizTalk Service | Custom Create.
Enter a unique name for the deployment, keeping the following values—EDITION: Developer, REGION: East US, TRACKING DATABASE: Create a new SQL Database instance.
In the next page, retain the default database name, choose the SQL server, and enter the server login name and password.
Note
As of writing this book, there can be six SQL server instances per Azure subscription.
In the next page, choose the storage account for archiving and monitoring information.
Deploy the solution.

The BizTalk Services create wizard from Windows Azure Management Portal

The deployment takes roughly 30 minutes to complete. After completion, you will see the status of the deployment as Active. Navigate to the deployment dashboard page; click on CONNECTION INFORMATION and note down the ACS credentials and download the deployment SSL certificate. The SSL certificate needs to be installed on the client machine where the Visual Studio SDK will be used.

BizTalk portal registration

We have one step remaining, and that is to configure the BizTalk Services Management portal to view agreements, bridges, and their tracking data. For this, perform the following steps:

Click on Manage from the Dashboard screen.
This will launch <mydeployment>.portal.biztalk.windows.net, where the BizTalk Portal is hosted.
Some of the fields, such as the user's live ID and deployment name, will be auto-populated.
Enter the ACS Issuer name and ACS Issuer secret noted in the previous step and click on Register.

BizTalk Services Portal registration

Creating your first BizTalk Services solution

Let us put things into action and use the deployment created earlier to address a real-world multichannel sales scenario.

Scenario description

A trader, Northwind, manages an e-commerce website for online customer purchases. They also receive bulk orders from event firms and corporates for their goods. Northwind needs to develop a solution to validate an order and route the request to the right inventory location for delivery of the goods. The incoming request is an XML file with the order details. The request from event firms and corporates is over FTP, while e-commerce website requests are over HTTP. Post processing of the order, if the customer location is inside the US, then the request are forwarded to a relay service at a US address. For all other locations, the order needs to go to the central site and is sent to a Service Bus Queue at IntlAddress with the location as a promoted property.

Prerequisites

Before we start, we need to set up the client machine to connect to the deployment created earlier by performing the following steps:

Install the certificate downloaded from the deployment on your client box to the trusted root store. This authenticates any SSL traffic that is between your client and the integration solution on Azure.
Download and install the BizTalk Services SDK (https://go.microsoft.com/fwLink/?LinkID=313230) so the developer project experience lights up in Visual Studio 2012.
Download the BizTalk Services EAI tools' Message Sender and Message Receiver samples from the MSDN Code Gallery available at http://code.msdn.microsoft.com/windowsazure.

Realizing the solution

We will break down the implementation details into defining the incoming format and creating the bridge, including transports to process incoming messages and the creation of the target endpoints, relay, and Service Bus Queue.

Creating a BizTalk Services project

You can create a new BizTalk Services project in Visual Studio 2012.

BizTalk Services project in Visual Studio

Creating the Order schema

From within your project, right-click on the project name, click on Add | New Item, and add a new Flat File Schema.

Add Flat File Schema into BizTalk Services project

Add the following nodes to the schema so that the structure looks as follows:

Flat File Schema structure

For each of the records in the XSD file, make sure that the delimiters are added correctly:

<b:recordInfo structure="delimited" child_delimiter_type="char" child_delimiter="," child_order="postfix" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" sequence_number="4" />

You can validate the schema by running it with an instance file. The Validate Instance command is available by right-clicking on the created schema file in the Solution Explorer. Add the following flat file and XML instances in two separate files, use the Validate Instance command, and verify that the schema validates those instances. For each command run, ensure that the schema properties window has the right Validate Instance Input Type (XML in this case):

OrderId|PaymentType|OrderDate|Code,Qty,Price,|Name,Email,Phone,|Recipient,Number,Street,City,State,Country,Pincode,|

<ns0:Order xmlns:ns0="http://BizTalkServicesOrderSample.OrderFF">
  <OrderId>MyOrder</OrderId>
  <PaymentType>CreditCard</PaymentType>
  <OrderDate>09-08-2013 22:50:00</OrderDate>
  <Product>
    <Code>100</Code>
    <Qty>1</Qty>
    <Price>500</Price>
  </Product>
  <Customer>
    <Name>Karthik</Name>
    <Email>user@hotmail.com</Email>
    <Phone>1-111-1111</Phone>
  </Customer>
  <ShippingAddress>
    <Recipient>Jon</Recipient>
    <Number>Building 1</Number>
    <Street>One Redmond Way</Street>
    <City>Redmond</City>
    <State>Washington</State>
    <Country>US</Country>
    <Pincode>98052</Pincode>
  </ShippingAddress>
</ns0:Order>

Tip

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

Creating the BizTalk Services solution

Open the bridge configuration surface (usually the MessageFlowItinerary.bcs file). The Visual Studio Toolbox should show the following entities:

Visual Studio Toolbox for BizTalk Services project

Use the VS toolbox to drag-and-drop FTP Source, Xml One-Way Bridge, One-Way Relay Endpoint, and Queue and connect them using the Connector, as shown in the following figure:

Bridge message flow in BizTalk Services project

Configure the following in the message flow:

Select the FTP server and configure the address, username, and password correctly.
Double-click on the bridge to open the Xml One-Way Bridge configuration:
- In the Message Types block, add the OrderFF.xsd instance created earlier.
  Bridge configuration
- In the first Enrich stage, add an XPath Type property reading from /*[local-name()='Order' and namespace-uri()='http://BizTalkServicesOrderSample.OrderFF']/*[local-name()='ShippingAddress' and namespace-uri()='']/*[local-name()='Country' and namespace-uri()=''] and writing to the location as a string. The XPath value can be obtained by opening the schema in VS, clicking on the relevant record, and copying the XPath value from the record properties window.
Promote property configuration in the Enrich stage of the bridge
In the parent MessageFlowitinerary.bcs view, click on the route link from OrderProcessingBridge to USAddressRelay and set the filter condition as location='US'; for the other link, set the location to US.
Route properties for the message flow
Edit the Queue .config file under MessageFlowitinerary.bcs and update the <tokenProvider> and the <endpoint> details with the Service Bus information.
Edit the Relay Service .config file under MessageFlowitinerary.bcs and update the <endpoint> details with the Service Bus relay information.
Build and deploy the solution.
If the deployment was successful, point your browser to https://<yourdeployment>/default/OrderProcessingBridge; you should see a 401 HTTP Error code stating a manage claim is required for this operation.

Filter reviews by

All

Amazon verified reviews

J.P.M. Smit Jun 24, 2014

The title of the book is getting started and that is what the book is about: getting a kick start to BizTalk Services. But it is more than just an overview. It touches all aspects of BizTalk Services and dives deeper in some chapters. So after reading this book you're not only familiar with BizTalk Services, but because of the step by step examples you're also capable of building your first BizTalk Services integration.The book is clearly written, touches the surface where applicable and dives deeper when necessary. And example is the B2B integration with EDI. This is a common scenario which can be implemented using BizTalk Services. The book dedicates an entire chapter to this.Recommended read.

Amazon Verified review

Steef-Jan Apr 09, 2014

Windows Azure BizTalk Services is a new service on the Windows Azure Platform. Released November 2013 and maturing fast through a three month release cadence to enhance and enrich its integration and B2B capabilities. Since the IT landscape is changing so rapidly over the last couple of years the integration needs have become more apparent. Connecting SaaS with on-premise is a common practice now and therefore having a more lightweight integration technology in the cloud makes sense. In case you are unfamiliar with this new paradigm, then this book will reveal this for you. It covers the Microsoft technology as is today and shows how to use the integration service to create solutions that run in a BizTalk Service. The book starts with what the service is and what the purpose of it is before it discusses the actual technology itself, explaining the concepts of bridges, EAI, B2B, messages, transforms, API, tracking, troubleshooting and migrating from BizTalk to its service counterpart. The latter is an interesting aspect as BizTalk and the BizTalk Service are not alike. Windows Azure BizTalk Service can be viewed as an integration Platform as a Service (iPaaS) or Cloud-based integration. This as a Service is fairly new and not a surprise with the increasing demand for integration. Cloud-based integration/iPaaS is characterized with a license model (pay as you go), deploy in the cloud, connectors, no code, and management/monitoring. And these characteristics will become visible with regards to the BizTalk Service if you read the book from cover to cover. It is an interesting technology and hopefully soon to be a key player in the iPaaS space with the current leaders like MuleSoft, Dell Boomi and others.

Kevin Smith Apr 01, 2014

“Getting Started with BizTalk Services” gives a unique perspective and understanding around BizTalk Services not only around the product but also how to leverage it in order to successfully deliver real world enterprise systems. The authors draw on their deep technical insights into BizTalk Services gained from building the product itself and from designing and implementing real world integration solutions using BizTalk Services.BizTalk Services represents the next evolution of the BizTalk Server platform, it’s a major shift from the previous on-premise architecture of BizTalk Server taking the platform to both cloud and hybrid models, as such there are some significant new concepts that engineers need to understand in order to be successful with the new platform. The authors do a great job of articulating this to the reader, the writing style makes the book approachable to both people new to the platform as well as seasoned BizTalk veterans wishing to up skill to the new platform. Jon and Karthik do a great job of guiding the reader through this exciting new technology. I was asked to review the book and found it an extremely useful resource in learning about BizTalk Services.

Peter Chang Jun 23, 2014

I'm so glad there is finally a book regarding WABS (Windows Azure BizTalk Services). Reading this book was so much easier and better way for me to get started on WABS. When it comes to learning a new technology, I often like to start with a sample/example, and this book starts with HelloWorld example. Then, it goes deep drive to each component and its concept. Since I've already worked with BizTalk Server, it was fairly quick for me to read the entire book, but this book was written in a way that even non-BizTalk people can easily understand. I would definitely recommend this book to others.

Dipti Porwal Jun 20, 2014

There are certain things one looks for while picking up a technical book. Firstly -content, second -approach and third- examples. "Getting started with BizTalk Services" book fares excellent in all the three criteria. It is a right fit to start with Integration on cloud using Microsoft technologies independent of one’s experience in Micrsoft BizTalk Server. An extremely comprehensive and pragmatic book that contains everything right from the Concepts, deployment basics, implementation, troubleshooting, WABS API to real world scenarios for both EAI and B2B. It even covers migrating to BizTalk Services from BizTalk Server. Tracking messages is an integral part of the messaging world and the same has been diligently explained in the chapter - "Tracking and Troubleshooting". I felt the authors have made themselves really accountable by writing this chapter as they have ensured that even the readers new to EAI can get pointers on troubleshooting the implemented scenarios and just not left them clueless in the maze of messaging components.Cloud is the order of the day and is gradually becoming integral part of today’s Enterprise. Even in the EAI space, it has become a mandate to understand and implement the integration scenarios on cloud. I would say, this is "The" book for getting started with BizTalk Services.

Getting Started with BizTalk Services: BizTalk Services offers great possibilities for bringing enterprises together in the cloud, and this book is the perfect introduction to it all. Packed with real-world scenarios, you will soon be designing your own tailor-made integration solutions.

What do you get with a Packt Subscription?

What you will learn

Product Details

What do you get with a Packt Subscription?

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

Filter reviews by

People who bought this also bought

FAQs