What is Camel?

This section provides a quick overview of what Camel is, and why it was created. Its goal is to help remind the reader of the core concepts used within Camel, and to help the reader understand how the authors define those concepts. It is not intended as a comprehensive introduction to Camel. Hopefully, it will act as a quick reference for Camel concepts as you use the various recipes contained within this book.

Integrating systems is hard work. It is hard because the developers doing the integration work must understand and how the endpoint systems expose themselves to external systems, how to transform and route the data records (messages) from each of the systems. They must also have a working knowledge of the ever growing number of technologies used in transporting, routing, and manipulating those messages. What makes it more challenging is that the systems you are integrating with were probably written by different teams of developers, at different times, and are probably still changing even while you are trying to integrate them. This is equivalent to connecting two cars while they are driving down the highway.

Traditional system integrations, in the way that we have built them in the past decades, require a lot of code to be created that has absolutely nothing to do with the higher-level integration problem trying to be solved. The vast majority of this is boilerplate code dealing with common, repetitive tasks of setting up and tearing down libraries for the messaging transports and processing technologies such as filesystems, SOAP, JMS, JDBC, socket-level I/O, XSLT, templating libraries, among others. These mechanical concerns are repeated over and over again in every single integration project's code base.

In early 2000, there were many people researching and cataloging software patterns within many of these projects, and this resulted in the excellent book Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe and Bobby Woolf, published by Addison Wesley. This catalog of integration patterns, EIPs for short, can be viewed at http://www.enterpriseintegrationpatterns.com. These patterns include classics such as the Content Based Router, Splitter, and Filter. The EIP book also introduces a model of how data moves from one system to another that is independent of the technology doing the work. These named concepts have become a common language for all integration architects and developers making it easier to express what an integration should do without getting lost in how to implement that integration.

Camel embraces these EIP concepts as core constructs within its framework, providing an executable version of those concepts that are independent of the mechanics of the underlying technology actually doing the work. Camel adds in abstractions such as Endpoint URIs (Uniform Resource Identifier) that work with Components (Endpoint factories), which allows developers to specify the desired technology to be used in connecting to an endpoint system without getting lost in the boilerplate code required to use that technology. Camel provides an integration, domain-specific language (DSL) for defining integration logic that is adapted to many programming languages (Java, Groovy, Scala, and so on) and frameworks (Spring, OSGi Blueprint, and so on), so that the developer can write code that is an English-like expression of the integration using EIP concepts. For example:

consume from some endpoint,
split the messages
  based on an expression, and
  send those split messages to some other endpoint

Let us look at a concrete example to show you how to use Camel.

Imagine that your boss comes to you, asking you to solve an integration problem for your project. You are asked to: poll a specific directory for new XML files every minute, split those XML files that contain many repeating elements into individual records/messages (think line items), and send each of those individual records to a JMS queue for processing by another system. Oh, and make sure that the code can retry if it hits any issues. Also, it is likely the systems will change shortly, so make sure the code is flexible enough to handle changes, but we do not know what those changes might look like. Sound familiar?

Before Camel, you would be looking at writing hundreds of lines of code, searching the Internet for code snippets of how best to do reliable directory polling, parsing XML, using XPath libraries to help you split those XML files, setting up a JMS connection, and so forth. Camel hides all of that routine complexity into well-tested components so you just need to specify your integration problem as per the following example using the Spring XML DSL:

<route>
  <from uri="file://someDirectory?delay=60000"/>
  <split>
    <xpath>/xpath/to/record</xpath>
    <to uri="jms:queue:myProcessingQueue"/>
  </split>
</route>

Wow! We still remember when we first saw some Camel code, and were taken aback by how such a small amount of code could be so incredibly expressive.

This Camel example shows a Route, a definition (recipe) of a graph of channels to message processors, that says: consume files from someDirectory every 60,000 milliseconds; split that data based on an XPath expression; and send the resulting messages to a JMS queue named myProcessingQueue. That is exactly the problem we were asked to solve, and the Camel code effectively says just that. This not only makes it easy to create integration code, it makes it easy for others (including your future self) to look at this code and understand what it is doing.

What is not obvious in this example is that this code also has default behaviors for handling errors (including retrying the processing of files), data type transformation such as File object to XML Document object, and connecting and packaging data to be sent on a JMS queue.

But what about when we need to use different endpoint technologies? How does Camel handle that? Very well, thank you very much. If our boss told us that now we need to pick up the files from a remote FTP server instead of a directory, we can make a simple change from the File Component to the FTP Component, and leave the rest of the integration logic alone:

<route>
  <from uri="ftp://scott@remotehost/someDirectory?delay=60000"/>
  <split>
    <xpath>/xpath/to/record</xpath>
    <to uri="jms:queue:myProcessingQueue"/>
  </split>
</route>

This simple change from file: to ftp: tells Camel to switch from using the hundreds of lines of well-tested code doing local directory polling to the hundreds of lines of well-tested FTP directory polling code. Plus, your core record-processing logic to split and forward on to a JMS queue remains unchanged.

At the time of writing, there are over 160 components within the Camel community dealing with everything; from files, FTP, and JMS, to distributed registries such as Apache Zookeeper, low-level wire formats such as FIX and HL7, monitoring systems such as Nagios, and higher-level system abstractions that include Facebook, Twitter, SAP, and Salesforce. Many of these components were written by the team that created the technology you are trying to use, so they generally reflect best practices on using that technology. Camel allows you to leverage the best practices of hundreds of the best integration technologists in the world, all in an easy to use, open source framework.

Another big innovation with Camel is that it does not require the messages flowing through its processing channels (routes) to be of any fixed/canonical data type. Instead, Camel tracks the current data type of the message, and includes an extensible data type conversation capability that allows Camel to try to convert the message to the data type required by the next step in the processing chain. This also helps Camel in providing seamless integration with your existing Java libraries, as Camel can type convert to and from your Java methods that you call as part of your Camel route. This all combines into an extremely flexible mechanism where you can quickly and easily extend almost any part of Camel with some highly focused Java code.

There is so much more to Camel than what can be covered in this very brief overview.