Integrating Systems

Using the techniques garnered in Chapter 3  Solution Fundamentals and Chapter 4  Data Streams, we can easily leverage Groovy to read data from one source, transform it in some manner (e.g., reformat it or perform calculations), then send it to another source. Think of this as using Groovy to thread systems together. Groovy, Java, and third-party libraries provide a range of functionality that aids in integrating systems. For example, libraries for working with key technologies such as HTTP, message queues, RESTful web services, data stores (relational and nonrelational), and FTP all exist. However, most of these libraries leave a fair bit of work for us to do.

Luckily, Apache Camel gives us a large library of integration components and a very readable approach to describing how data is routed. Take a look at Code Listing 71: Example Camel Data Route and you'll see some of the key components for defining how data enters the route, is processed, and creates an output:

• The from method configures the source of the data for the route. The string passed to the from method describes how data is consumed—in this example, I use the file component to read files from the data_in directory. In fact, the code we'll see soon watches this directory for new files and processes them as they appear.
• Once a file has been read, I call unmarshal to set up an object that will hold the incoming data. In this case, I'm reading CSV data into a bean I've prepared.
• Next, I’m able to marshal the data into JSON format.
• And finally, I save the JSON data into a file in the data_out directory. I base the new file name on the original by using the that file's name (without its extension) and add the .json extension.

Some of these items above won't make a lot of sense outside of the full code listing, but hopefully you can see that Camel is dealing with all of the tricky aspects around handling the files, monitoring for new data, and converting formats.

Code Listing 71: Example Camel Data Route

Notice that copying the previous code into Groovy Console won't yield a result. First, we need to put up some scaffolding to support the Camel route. So let’s go through setting up Camel routes that work with files, message queues, and databases.

Working with files

Camel is based on Enterprise Integration Patterns (EIP), which is described well in the book Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf. Because many of our integration requirements have been around for some time, we don't need to write our own solution for this type of text and Camel's implementation—we need only to configure the framework to meet our local needs.

In the File Transfer pattern, one application exports data into a file that is imported into another application. Camel supports this pattern through a number of components:

• The FTP and FTPS components support sending and receiving files through the File Transfer Protocol.
• The Jetty component can consume and produce HTTP-based requests.
• The File component supports reading and writing files in local directories.

Let’s work with the File component, as it's straightforward and easier to pick up. The File component will poll a directory and wait for new files to be added. Once a new file appears, it will be sent to the route and processed. The File component can also write files to a directory. This allows us to set up a route that polls a directory for a CSV file, process it in some manner, then store the result in a new directory.

Before we set up the route, we first need to prepare a class that can hold the incoming CSV data. We can imagine that such data might be produced by a data logger that measures the current temperature. Each measurement is recorded against the date and time (timestamp) of the recording and saved in a CSV file stored in a specific directory. Over time, this directory will contain a series of CSV files, each one containing a single line that looks something like: 2016-02-21T09:40:28.922,916.

Code Listing 72: A Bean for Data Logging describes a class, DataEntryBean, that holds the two properties created by the data logger:

• The timestamp property is held as a String.
• The value property is held as an Integer.

Both properties have been marked as final so that, once we've created a DataEntryBean object, the properties can't be changed.

DataEntryBean provides a no-argument constructor, DataEntryBean(), that sets timestamp to the current date and time and generates a random number between 0-999 for the value property. Perhaps it's a very hot planet.

The toString() method returns a String representation of the object. You’ll see that I separate the properties with a comma—handy for my CSV needs. Groovy methods and closures will return the result of their last expression, so I don't need to explicitly use the return keyword.

Hopefully, most of the syntax for creating a class in Groovy is self-evident. However, the various annotations may look a bit odd. First of all, the @CsvRecord and @DataField annotations are from Camel's Bindy component. Bindy helps us work with CSV data (among others), and the @CsvRecord annotation indicates that each DataEntryBean represents a line in a CSV file. I pass the annotation two arguments: the separator used in the CSV file and whether or not the first line in the CSV is to be skipped. This second argument lets us ignore a header row if need be.

The two properties (timestamp and value) carry a @DataField annotation that indicates that the property aligns with a field in the CSV. Each annotation indicates the position of the field, that we require it to be present, and that any leading/trailing whitespace should be trimmed.

Code Listing 72: A Bean for Data Logging

The DataEntryBean class also features a number of XML annotations, and these allow us to bind to an XML structure. We'll explore this in more depth shortly.

Now that we have a bean to hold CSV data, let's create a script that will fake a data logger for us. Code Listing 73: A CSV Producer will generate a new CSV file every second and place it into the data_in directory.

Code Listing 73: A CSV Producer

First of all, GrabConfig is set up in the same manner we set it up in the section on Databases because it lets Camel find the component classes. Next, we Grab the camel-core library and import two classes:

• Main: provides an easy way for us to get a Camel-based script up and running.
• RouteBuilder: provides the basis for building a route that looks like Code Listing 71: Example Camel Data Route. 

Note: You'll notice that I've imported the Logback library (logback-classic). This sets up logging for the script and is used by Camel. An associated file, logback.groovy, can be found in the source and is used to configure the logging.

We next use a handy piece of syntax to create what's called an anonymous class: new Main() {..}. This essentially extends Camel's Main class on the fly. Within the curly braces ({}), we can configure the anonymous class and, once that's setup, call the run() method to start up Camel.

Here, I have performed a few tasks within the anonymous class. The easiest task to see is where I override the Main class's afterStart() and beforeStop() methods to log some useful messages.

Note: In order to avoid repetition, I won't restate the afterStart() and beforeStop() code in the remaining code listings. The full code is available in the source code accompanying this book.

Somewhat more perplexing is the block of code surrounded by another set of curly braces. This is an instance initializer block and is run whenever a new instance of the class is created. Because I'm using an anonymous class, this is run straight away. The block itself starts with a call to the enableHangupSupport() method because this allows the route to be gracefully stopped using Ctrl+C. Next is the call addRouteBuilder new RouteBuilder() {..}. This is a method call (addRouteBuilder) for setting up a Camel route. The argument to addRouteBuilder is another anonymous class, this time extending the RouteBuilder class.

Within the RouteBuilder class, we need only to override the configure() method, and this is where we set up the Camel route. I've broken out the route into Code Listing 74: The Fake Data Logger Route to help us see what's happening. I use the timer component to trigger every one second. In Camel, we are concerned with endpoints, and these sit at either end of a route and consume or produce messages/data. An endpoint is described using a URI that indicates the required component (e.g., timer: for the timer component) followed by the context path (e.g., the name of the timer—demoTimer) and, finally, the options for the endpoint (e.g., the timer period—period=1s). The timer endpoint is passed into the from method and fires off the route every second.

We use the process method call to define a processor that can manipulate a message. I won't get too deep into Camel's structure here, but Camel works on an exchange container mechanism in which there is an "in" and an "out" message within the container[5]. This is why the closure passed to the process method declares one parameter named exchange. As we have no real source message (just a timer event), the closure only creates a new DataEntryObject (whose constructor will create default values) and adds this object's string representation to the exchange's out message.

We use the to method to define another endpoint. In this case, I use the File component to save the message contents into a file in the data_in directory. I set the filename to the current date-time with a .csv extension. It is important to note that, while the URI contains the ${} syntax, this isn't a Groovy String—it's using Camel's Simple Expression Language. Happily for us, Camel will create the destination directory if it doesn't already exist.

Code Listing 74: The Fake Data Logger Route

Now that we've worked through the code, how do we run it? I'd suggest that we run it from the command line because stopping it is easier than with using the Groovy Console. Start by changing into the camel directory of the sample source code for the book. Then start up the script with groovy ProducerFile.groovy and you'll see Camel prepare its configuration and start producing output every second (e.g., INFO camelApp - Exchange[Body: 2016-02-27T13:47:51.017,158]). This means that Camel is now creating a series of CSV files in the data_in directory. This will keep going until you press Ctrl+C, which will shut down Camel

Note: If you check out the ProducerFile.groovy file, you'll see that it has a little extra code—specifically an extra endpoint that outputs log entries.

Once you shut down the script, check out the data_in directory and you'll see a fresh set of CSV files.

CSV to JSON

Now that we have created a fake data logger that generates CSV files, let's create the route for importing the CSV data. Code Listing 75: CSV to JSON Convertor sets up Camel in the same manner we've just seen but with a route configured for reading a CSV file and converting it into a JSON file. The data_in directory is monitored by the file component and forms the route's inbound endpoint.

The call to unmarshal allows us to take the incoming comma-separated record (e.g., 2016-02-27T13:38:15.891,317) and convert it to a new data format. In this example, we call on the camel-bindy library (note the import alias for BindyCsvDataFormat) to set up a DataEntryBean object with the data from the CSV file. This is where those @DataField annotations used in DataEntryBean come into play. These guide Bindy in allocating CSV fields into Groovy object properties.

Once the data is within the bean, we can marshal it into JSON format through the calls marshal().json(true). This uses the camel-xstream library to create a JSON representation of the data. The JSON is printed by passing true to the json method.

Finally, the File component is used to place the resulting JSON into a file stored in the data_out directory.

Code Listing 75: CSV to JSON Convertor

Running groovy ConsumerFileJson.groovy will cause the CSV files in data_in to be read, and the equivalent JSON format will be saved into the data_out directory. As Camel processes the files from data_in, it will transfer those files into the data_in/.camel directory and won't be read again. Code Listing 76: Example JSON Output provides an example of a generated JSON file.

Code Listing 76: Example JSON Output

CSV to XML

Converting the CSV into XML doesn't require much more work than converting to JSON. First, the DataEntryBean[6] is decorated with a number of annotations:

• @XmlRootElement is used to map the DataEntryBean to an XML element.
• @XmlAccessorType(XmlAccessType.FIELD) indicates that all of the DataEntryBean fields (properties) will be mapped to XML.
• @XmlAttribute is used for each property in order to indicate that the DataEntryBean property will become an attribute rather than a subelement.

The Java Architecture for XML Binding (JAXB) provides these annotations.

We can now map a DataEntryBean to XML, and the camel-jaxb library will take care of the details. Take a look at Code Listing 77: CSV to XML Convertor—it looks much the same as the CSV to JSON version, but with the marshal call changed to marshal().jaxb(true) and the resulting file stored as XML.

Code Listing 77: CSV to XML Convertor

Tip: You might need to run groovy ProducerFile.groovy again if you don't have any CSV files waiting in the data_in directory.

Running groovy ConsumerFileXML.groovy will cause the CSV files in data_in to be read, and the equivalent XML format will be saved into the data_out directory. Code Listing 78: Example XML Output provides an example of a generated XML file.

Code Listing 78: Example XML Output

Although the CSV to JSON/XML examples have been based on local files and directories, it's not a big leap to setting up routes to CSV files created in one application and made available via FTP or HTTP.

Note: For the adventurous, why not run the ProducerFile.groovy script in one console and one of the ConsumerFile scripts in another? You'll see the producer creating records and the consumer picking them up.

Working with queues

Moving on from the File Transfer pattern, let's look at how we can use Camel to place messages on a message queue and read the messages from the queue. This is the basis of the Message Channel EIP.

Let's take a moment to look at message queues. In Figure 2: High-Level View of a Message Queue, you'll see four main elements:

• The Producer adds Messages to the Message Queue.
• The Message is data—perhaps text (e.g., XML or JSON)—that we're sending from the Producer to the Consumer.
• A messaging system manages the Message Queue.
• A Consumer takes messages from the queue and, presumably, does something with the message.

Figure 2: High-Level View of a Message Queue

There are many benefits to message queues, including asynchronous processing, which most developers prefer. Asynchronous processing allows the producer to add messages to the queue without relying on the consumer being active/ready. Each new message waits in the queue until the consumer is ready to handle them. If you’ve ever used email, then you've used queues—other people (producers) send you emails, and the messages wait on the mail server (message queue) until you (the consumer) are ready to read them.

Before we can use Camel with messaging, we'll need to set up an Apache ActiveMQ messaging server. ActiveMQ will manage the message queue and Camel will both produce and consume messages.

Installing Apache ActiveMQ

The ActiveMQ Getting Started guide is the best place to go for installation instructions, but the steps below should get you going quickly:

1. Open the Apache ActiveMQ download page: http://activemq.apache.org/download.html.
2. Download a release version (such as ActiveMQ 5.13.1 Release) for your operating system (Windows or Unix/Linux).
3. Extract the downloaded archive file.
4. Change into the new directory.
5. Start the server with:

1. Windows: bin\activemq start
2. Unix: bin/activemq console

6. Access the web-based administration interface http://0.0.0.0:8161/admin/ with:

1. Username: admin
2. Password: admin

If you have problems getting the server to start, check out data/activemq.log for log messages.

Once the server is running, try out the following commands:

• Start the consumer in one terminal: bin/activemq consumer --messageCount 100 --destination queue://TEST. This will consume new messages in the TEST queue and cease once 100 messages have been consumed.
• Send a message from a producer from another terminal: bin/activemq producer --message "Hello,\ world" --messageCount 100 --sleep 1000 --destination queue://TEST. This will generate a new "Hello, world" message every second and cease once 100 messages have been sent.

To clear the queues, run bin/activemq purge while the server is running or use the admin interface.

After you make sure that ActiveMQ is running, you're ready to start sending and receiving messages.

The message producer

Sending messages to the message queue isn't overly different from the code we wrote to create new CSV files[7]. Code Listing 79: Message Producer is based on a timer endpoint and the same process we'd seen previously—every second we create a new DataEntryBean object. In order to communicate with the message server, we use the ActiveMQ component, so that defining the endpoint is easy: 'activemq:DEMO'. The DEMO context path designates the queue to which messages will be sent. Happily, ActiveMQ will create this queue for us automatically.

That's all we need to get messages to the server. Just run groovy ProducerQueue.groovy and you'll start to see the messages being delivered. To make sure that it's working, open the ActiveMQ web interface and browse to the DEMO queue—the URL will look like this: http://0.0.0.0:8161/admin/browse.jsp?JMSDestination=DEMO. You should see a list of messages, and you can click on an individual message to see its details.

At this point, stop the ProducerQueue.groovy script. The DEMO queue will have the messages waiting for us to consume them.

Code Listing 79: Message Producer

The message consumer

Consuming the DEMO queue messages is easy with Camel. As you can see with the route defined in Code Listing 80: A Basic Message Consumer, all we need is from('activemq:DEMO').to('stream:out'). The ActiveMQ endpoint picks up messages in the DEMO queue, and our route sends them to the console using the camel-stream component. This component allows us to write to the standard output stream—in this case it outputs to the console.

If you now run ConsumerQueueStream.groovy, you'll see the messages being read and then displayed to the screen (e.g., 2016-02-27T15:55:28.978,655). The script will pick up all of the messages we created earlier and, if you go into the ActiveMQ web interface, you'll see that the queue is now empty.

Code Listing 80: A Basic Message Consumer

That consumer was quite basic so, for our final Camel script, let's consume messages and insert the data into a database. This route is one you're likely to have seen a number of times, and we can imagine a weather sensor feeding data into our system, so we'll send the data to a message queue for another system to collect and add to a database.

The script is a little more involved for defining a route that consumes messages and inserts the data into a database. Code Listing 81: Storing Message Data in a Database isn’t too different from what we've already looked at, but there are a few noteworthy elements. First, in order to use the JDBC component (to('jdbc:demoDb')), we need to define the demoDB connection. This isn't as straightforward as simply adding the JDBC connection URL as an option in the endpoint definition. Instead, we need to register the connection in a registry accessed by Camel. While we could use JNDI or Spring, let’s use the registry made available as a field in Camel's Main class. To define the data source (an SQLite database):

1. Create a BasicDataSource[8] object: def ds = new BasicDataSource().
2. Set the database's JDBC URL: ds.url = 'jdbc:sqlite:logger_data.db'.
3. Add the data source to the registry field: registry.put('demoDb', ds).

You'll note that the name I give to the connection (demoDB) matches the one used by the JDBC endpoint.

Before we declare the route, we’ll use Groovy's SQL library in order to make sure the required table exists in the database.

The JDBC component expects that the message body will contain the SQL required to perform the action needed by the route. There are several effective ways to do this, but I've decided to demonstrate the use of a bean to prepare the SQL. In Groovy, a bean is essentially a Plain Old Groovy Object (POGO), and the LoggerDbInsert is a very basic class with one method. While I've declared this class within my script, you can use one from another library—just make sure it's available to Groovy (e.g., with @Grab).

In order to use my bean in the route, I simply include .bean(LoggerDbInsert) in my Camel route. You'll note that I pass the bean method the name of the class but not an instance/object. Camel has the smarts to locate the class, create an instance, and call the insertRow method with my DataEntryBean object created from the CSV data in the incoming message. This is easier because the LoggerDbClass has only one method, but note that I could have specified the method as a second argument to the bean call.

Tip: There is a bean component, and I could have used this in a .to() call, but .bean() is straightforward.

Code Listing 81: Storing Message Data in a Database

So, with a little work to define the connection and add it to the registry, combined with providing a basic class for setting up the SQL Insert command, my Camel route is able to read data from the queue and insert a new row into the database. As I mentioned earlier, there are other approaches to this, and I've included two examples in the source code: ConsumerQueueDatabaseSetBody.groovy and ConsumerQueueDatabaseProcess.groovy.

Summary

This chapter has introduced you to using Apache Camel in Groovy scripts, which, as you’ve seen, is a powerful combination. Take some time to browse the list of Camel components and you'll find support for a wide range of systems. Camel can make it easy to try out new systems, and I hope that these examples give you a good starting point.

My example code is written as Groovy scripts and can be run as stand-alone elements. This approach can assist many solutions and may be enough for your project. However, you may wish to explore more substantial middleware solutions, and you'll be happy to learn that Camel can work with many of the existing platforms. Take a look at Apache ServiceMix, JBoss Fuse, or Camel's support for the Spring Framework if you're interested in a larger, enterprise approach.