Solution Fundamentals

---

In Chapter 2  Language Essentials, we covered enough of the key Groovy syntax that you'll be able to read through Groovy code and get an inkling of what it is doing. In this section, we'll build on that understanding and bring in a range of libraries that will help you get the job done.

Libraries

Java provides a wide range of code libraries that Groovy can easily access. The built-in Java libraries are referred to as the Java API (Application Programming Interface), and you'll find them documented in the API Specification. The first thing you'll see is a list of packages such as java.lang and java.time. Packages provide a method of grouping similar classes so that they can be easily located. If you find yourself writing larger applications in Groovy, you'll want to start arranging your code into packages, and the Java Tutorial can help you with this.

Note: Code that doesn't declare itself to be in a package (such as our scripts) is allocated to the default package.

Groovy modifies some of the Java libraries so as to make them Groovier—this is referred to as the Groovy Development Kit (GDK). The GDK is documented on the Groovy website, and it's worth looking at the GDK documentation prior to checking the Java API's because you'll see where Groovy gives you extra functionality. The Groovy API provides another set of libraries that really let us flex Groovy's versatility.

Tip: Several classes in the org.codehaus.groovy.runtime package add in a number of methods to Java classes. Checking out documentation for classes such as StringGroovyMethods, DefaultGroovyMethods, and the typehandling subpackage can help when you are not certain where a method came from.

Outside of the core Java and Groovy libraries, you'll find a vast array of existing libraries that people have written to overcome various problems. These libraries are offered through open source or commercial licenses and will often spare you from having to write a lot of a codebase yourself. Finding a useful library can be tricky for newcomers, but here are a few sites that will help you get started:

• Maven Central is a key repository for Java libraries and the default for a number of tools.
• MVNRepository provides an alternative interface for searching Maven Central.
• Bintray's JCenter provides a collection of libraries from across a number of repositories, including Maven Central.

These repositories provide a basic model for describing a library using three identifiers:

• groupId: designates a project or an organization.
• artifactId: designates a specific library from the project/organization.
• version: designates the version of the library.

We use a library’s coordinates when we want to include the library in our code. The coordinates use the following pattern: groupId:artifactId:version. The coordinates allow us to be specific about what we want to use. For example, we'll look at the Apache Commons CSV library shortly, and the coordinate of org.apache.commons:commons-csv:jar:1.2 specifies version 1.2 of that library.

Once you know the coordinates, using libraries is easy in Groovy scripts, thanks to the Grape dependency manager. Grape provides the Grab annotation that allows us to include the Apache Commons CSV library in our scripts through a one-liner: @Grab('org.apache.commons:commons-csv:1.2'). This tells the Groovy compiler to download the requested library and make it available to your code.

Note: Annotations perform a variety of functions within code. The ‘at’ symbol (@) is used when calling the annotation.

The import keyword is used to make library classes available to our code. You'll see this used in most of the code listings, and you’ll soon get the hang of the usage, but let's look at a few examples:

• import java.nio.file.Paths: will import the Path class from the java.nio.files package.
• import java.nio.file.*: imports all classes in the java.nio.files package.
• import static org.apache.commons.csv.CSVFormat.RFC4180: will import a class field (RFC4180) from the (CSVFormat) class and allow us to use it directly.
• import static java.nio.file.Paths.get as getFile: imports a class method (get) from the Paths class. The get method could be available to call directly in our code, but I've set an alias (getFile) to use instead (I think that "get" is a little too generic.)

Groovy imports the following packages/classes by default:

• java.lang.*
• java.util.*
• java.io.*
• java.net.*
• groovy.lang.*
• groovy.util.*
• java.math.BigInteger
• java.math.BigDecimal

There's no harm in explicitly importing those default items but, over time, you'll find that you simply take them for granted.

Next, I’ll demonstrate how to use a range of existing Java and Groovy libraries in your code.

Reading and writing data

Data comes to us from a range of sources such as files, web resources, and databases. Groovy can read and write these sources with ease.

The source code for this chapter is located in the fundamentals directory of the source code.

Files

Java's java.nio.file package provides a range of classes for working with files. The Path class, which represents an object in a file system, is central to this approach. Path objects help us write code that doesn't get too caught up in how various operating systems manage files, which is essential for a cross-platform system. Groovy extends Java's Path class so that there's even less to do.

We get an introduction to the sequence of a number of concepts in Code Listing 31: Reading a File:

1. In order to access a Path object, we call the get class method provided by the Paths class:

1. In order to use the Paths class, we import it using import java.nio.file.Paths.
2. The call to Paths.get is passed the name of the desired file ('demo.txt') as an argument and returns a Path object that is assigned to the file variable.

2. The code then demonstrates two approaches to reading the file:

1. The text property contains all of the text in the file.
2. The withReader method uses a buffer for reading a file's content, which is handy with large files. The call to withReader is passed a closure that is used to process the file. Once the closure has completed, the file is closed.

Code Listing 31: Reading a File

When looking at this code, you might find it helpful to refer to the documentation for the various classes:

• The Java API documentation for the Paths and Path classes.
• Groovy's supplements to the Path class, including the withReader method.

If you flick through the documentation, you'll notice that Path has no text property. However, Groovy enhances Path with the getText method, which leads to an interesting "trick" provided by Groovy: a method starting with get that has no parameters can be read as a property, as we saw in file.text. The name of the property is determined by dropping the "get" prefix and using lowercase for the first character in the remaining string.

This approach utilizes a common Java convention around getters and setters. Traditionally, the method to read an object's field will use the "get" prefix followed by the name of the field. An associated setter is the provided to change the value of a field and uses both the "set" prefix and a single parameter for the new value.

Tip: When you start writing your own classes, you'll find that Groovy can automatically provide getters and setters, which will save your code from a huge amount of boilerplate.

Code Listing 32: Writing a File demonstrates the use of a setter as a property by writing a haiku to a file through a very easy statement: Paths.get('haiku1.tmp').text = haiku. This statement gives us access to the haiku1.tmp file (even though it doesn't exist), then set its content through the text property. Almost too easy. The code also demonstrates two other approaches:

• Using the withWriter method that, much like withReader, uses buffering to help with larger files and closes the file once the closure has completed.
• Using the Files class to create a temporary file to which you can write.

The Files class provides a number of useful class methods for working with filesystems, including:

• Uses exists, which determines if a file exists.
• Checks for filesystem objects:

• isDirectory
• isExecutable
• isReadable
• isWritable
• isRegularFile

While the examples provided in this e-book don't perform checks such as Files.exists, you will probably want this in production code.

Code Listing 32: Writing a File

Code Listing 33: Appending a File demonstrates adding to a new or existing file. I've taken a slightly different approach to Paths.get by using import static, which allows for a specific class member such as the get method to be made available directly in the code. By doing this, I could have called get('pets.tmp'), but I decided to alias the method by using as getFile in the import. I chose this option solely because it makes the method a bit clearer in a larger script.

Once the getFile alias has been established, I can add items to a file in one of two ways:

• Through the use of withWriterAppend.
• Through the use of the << operator to append in much the same was as appending to a list.

Code Listing 33: Appending a File

Web resources

The toURL method available on strings makes working with URLs straightforward. Code Listing 34: Reading from a URL demonstrates this by converting a string value ('http://www.example.com/') to a URL, then accessing the contents through the text property—just as we read a file's contents. The alternative use of the withReader method can be useful for larger web pages.

Code Listing 34: Reading from a URL

We should note that the previous code accesses a page available at a website—it doesn't access a web service. If you'd like to access a web service, take a look at groovy-wslite or Jersey.

Databases

It’s easy to feel as if there are as many libraries for working with databases as there are database products. Java's JDBC API has been around since JDK version 1.1 was released in 1997. It’s a pretty direct approach to working with relational databases, and it generally requires the direct use of SQL. Over the years, a number of libraries have appeared to handle not only relational data but other data structures as well. This work has been annexed under the concept of "persistence" and, for Groovy developers, the various persistence libraries help bridge the gap between Groovy objects and persistence. The following libraries/APIs are worth exploring as your experience grows:

• Grails' Object Relational Mapping (GORM): for relational and nonrelational data.
• Java Persistence API (JPA): for mapping objects to relational data.
• Java Data Objects (JDO): for mapping objects to relational and nonrelational data.

Those technologies are useful in larger applications, but their level of abstraction can be cumbersome in small codebases and scripts. I'll keep it simple and lean towards working with direct SQL queries. Groovy's groovy.sql package provides enough of a layer over JDBC to make it easy to work with databases without getting quite as involved as working with JDBC.

I'll utilize an SQLite database for the first two examples because providing you with a sample database file (see data/weather_data.db in the source code) is easier that way and doesn't require you to set up a server. If you'd like to explore the sample database in a graphical tool, I suggest you grab a copy of the DB Browser for SQLite.

Working with a database requires the use of the appropriate JDBC driver. I've chosen to use the Xerial SQLite driver because it includes all of the components required to interact with an SQLite database. In order to utilize the driver in a Groovy script, we'll need to @Grab the coordinates org.xerial:sqlite-jdbc:3.8.11.2. We'll also need to use a configuration setting to tell Grape that it should attach the driver to our script. That last sentence is rather vague because the specifics aren't well-placed for this e-book. Just know that you should always include the following line of code when using JDBC drivers in your scripts: @GrabConfig(systemClassLoader = true).

Once you have a JDBC driver loaded, you can connect to the database using a JDBC connection URL. These follow the general format jdbc:driverName:databaseName;properties. For accessing an SQLite database, we use jdbc:sqlite:databaseFile where databaseFile is the location of the SQLite database. The example code for this chapter is in fundamentals/database/, and the database file is in the relative location of ../../data/weather_sqlite.db, which results in a connection URL of jdbc:sqlite:../../data/weather_sqlite.db. Given this URL and the @GrabConfig setting, Groovy will locate the driver and we can start querying a database.

With that configuration overview complete, let's turn to Code Listing 35: Querying a Database and see how it comes together. The first two lines configure Grape and grab a copy of the JDBC driver[2]. Once we have the driver, we import the groovy.sql.Sql class and call the Sql class’s static withInstance method. This method takes two arguments: the JDBC connection URL and a closure that can work with the database connection. Instead of simply using it within the closure, we declare the sql parameter to aid in readability.

I display a bit of text within the closure in order to explain what's being queried. The call to sql.eachRow runs a query (the first argument) and passes each row of the result set to a closure (the second argument). I've written the query within a Groovy String so that I can pass in a variable and rely on the Sql class's protection against SQL injection. The closure receives one row at a time, and accessing the individual column values is easy—we use the dot-point notation seen in maps (e.g., row.recordingDate).

Code Listing 35: Querying a Database

Code Listing 36: Querying a Database (version 2) follows the same pattern as Code Listing 35, but the SQL query calculates the annual average rainfall. We don't need a Groovy String because we don't need to insert a variable, so we simply use the fixed-string form. The average may have any number of decimal places, so we should round the output using round(2).

Code Listing 36: Querying a Database (version 2)

Data formats

Comma-separated Files (CSV)

The comma-separated file is much derided, but it never seems to disappear (most likely due to its alignment with relational databases and its readability). Unfortunately, there's no real CSV standard, and you'll find that systems will differ on the separator they use, the role of quotes, and the use of line breaks. For these examples, we’ll use the CSV model described in RFC 4180 and rely on the Apache Commons CSV library because it can work with RFC 4180 format as well as others commonly labeled "CSV."

As you look at Code Listing 37: Reading a CSV File, you might recognize the first few lines:

• I've imported the Paths.get method with the getFile alias.
• I use Grape to get a copy of the Apache Commons CSV library. From this, I import the RFC4180 class that predefines the CSV format I want to use.

Next, the code gets a little more complex than the examples we've seen so far. I'll break it down into steps:

1. RFC4180.withHeader() returns a CSVFormat object that can read an RFC4180-compliant file in which the first row contains the column names.
2. In .parse(getFile('../../data/weather_data.csv').newReader()):

1. The call to getFile loads a Path to the CSV file, then newReader opens a reader on the file.
2. The reader is used by the CSVFormat’s parse method to correctly parse the rows in the CSV file, which will return a CSVParser object.

3. Once the CSV has been parsed, I call the iterator() method, which will let me iterate through the rows using the each method.

We've seen the each method previously and, in this case, the closure is passed one row from the CSV at a time. As the column headers in a CSV may contain punctuation, Groovy's ability to access map keys using strings comes in handy when accessing the rainfall field: record.'Rainfall (millimetres)'.

I've also used the with method to make the code a bit more readable. Groovy enhances Java's Object class (the basis of all Java classes) by adding the with method, which accepts a closure argument. The code within the closure is invoked against the calling object in the first instance. This means we can reduce the verbosity down from println "$record.Year-$record.Month-$record.Day: $rainfall".

Code Listing 37: Reading a CSV File

This approach of using a series of method calls is referred to as method chaining and relies on each method to return an object on which another method can be called. I encourage you to use this approach because it avoids the need for temporary variables, which makes code more readable. To help you see how method chaining works, I'll use this style of programming for the rest of the e-book.

JavaScript Object Notation (JSON)

The JavaScript Object Notation (JSON) has become ubiquitous in the world of data exchange, configuration files, and in nonrelational data storage. Groovy makes reading JSON files easy through its JsonSlurper class. In Code Listing 38: Reading a JSON File, the contents of a JSON file are read into the script, then handled as we would treat any other Groovy list.

The code reads the JSON file (weather_data.json) using withReader, much as we've seen previously. Within the closure, an instance of JsonSlurper is used to parse the file's contents and place the resulting data structure (a list of objects) into the weatherData variable. At this point, we have a variable containing an easily navigable version of the JSON data.

The call to weatherData.findAll[3] takes a closure argument that provides a filter for each entry—in this case we want all records for August 2015. All entries in weatherData that match this filter are returned in a list, and against this we call the each method to display the rainfall details.

Code Listing 38: Reading a JSON File

Groovy makes it easy to read JSON, but how about creating JSON data? Yup, Groovy has you covered. In fact, it's so easy that I'm going to focus the next example on building on the findAll aspect of the previous example.

Code Listing 39: Preparing JSON Output demonstrates an approach to reading a CSV file and producing JSON. Much like we saw in the CVS chapter, the code reads in the CSV file, then gets an iterator to read through the rows. I've used the iterator's findAll method to get all entries for January 2010. The subsequent call to the collect[4] method is used to return a transformed set of records. In this example, I have used collect to create a map for each record that will simplify field access and create a convenient date string for the day field. This is a simplistic use of collect to remap the data—I could have performed calculations or other, more involved, processes.

Once we have the list of objects, we need only to call JsonOutput.toJson to convert the data into JSON. To make this a little more pleasant to read, let’s next call JsonOutput.prettyPrint to display a nicely formatted version.

Code Listing 39: Preparing JSON Output

The partial JSON output from the previous example is illustrated in Code Listing 40: JSON Snippet. 

Code Listing 40: JSON Snippet

Extensible Markup Language (XML)

Just as you can slurp up a JSON file, Groovy lets you slurp up XML. Code Listing 41: Reading an XML File demonstrates the use of the XmlSlurper class to parse an XML file. Just as with the JSON example, I get a reader for the XML file. Within the reader's closure, I create a new instance of XmlSlurper on the fly and call the parse method to read the XML data. The weatherData variable now holds a GPath object that we can use to interrogate the XML. It's handy to know what's in the XML, and I've included a snippet in Code Listing 42: XML Snippet.

Code Listing 41: Reading an XML File

Code Listing 42: XML Snippet

Now that we’ve parsed the XML data, let’s get a subset of records, in this case the weather readings for August 2015. In order to set up this filter, we first create a date variable (dateCheck) using the YearMonth class from the java.time package (a very useful one to know). This package also includes a Month enumeration, which we use in order to get a "constant" for AUGUST.

Once the dateCheck has been set up, we call weatherData.reading.findAll because weatherData holds the root node of the XML file, and we want to findAll instances of the reading element that relate to August 2015. The closure passed to findAll is supplied with each instance of the reading element, and the closure performs a check against the reading's date. This looks a little messy at first, but I'll break down the sequence for you:

1. reading.@day accesses the day property in the reading element.
2. [email protected]() returns the value of the property (e.g., 2006-01-01).
3. [email protected]()[0..-4] returns a substring of the value, starting at index 0 through to the 4^th-last character (e.g., 2006-01).
4. YearMonth.parse([email protected]()[0..-4]) creates a type of date object representing a year and month combination.

This would look like YearMonth.parse('2006-01') for one of the readings in Code Listing 42: XML Snippet.

When the year-month date object has been determined, we can check it against dateCheck to determine if the reading date meets our needs. Once findAll has completed, we will have the subset of XML elements that meet our requirement (readings from August 2015), and we can call on the each method to display them. Remember that each is called against those XML elements, so we must use @day.text() and @rainfall.text() to access the element's property values.

In the section on Web resources, I demonstrated the use of the toURL method to create a URL object from a string and call the text property to download a resource. Once downloaded, this is just text (a string) that can be read using XmlSlurper's parseText method. Code Listing 43: Reading XML from a URL demonstrates this process and displays the title and updated element values from the downloaded XML. We can use a similar approach to accessing JSON content from various web resources.

Code Listing 43: Reading XML from a URL

Tip: This works for XHTML pages, but check out http://jsoup.org/ if you want to parse HTML pages.

Now we’re into creating XML and Groovy, which gives us the MarkupBuilder that makes this particularly easy. Groovy's builders are based on hierarchical object structures such as the ones we see in XML, HTML, JSON, graphical interfaces, etc., and Groovy makes full use of its dynamic nature in order to help the developer build these structures easily.

For XML work, using Groovy's MarkupBuilder lets us avoid swaths of method calls to create elements and properties while keeping the logic inside the code (rather than using templates).

Code Listing 44: MarkupBuilder for XML

Code Listing 44: MarkupBuilder for XML contains this sequence of familiar code:

1. We get a writer for a new XML file with a minor addition—we want to make sure that the file is UTF-8 encoded.
2. We then create a new MarkupBuilder object and store it in the weatherData variable.
3. Calling weatherData.rainfall, we next create the structure we need.
4. To complete the script, we close the writer.

The output of the script is provided in Code Listing 45: MarkupBuilder for XML—Output, and it all looks pretty good to me. However, take another look at item 3 in the explanatory list above. It looks wrong. After all, how did the MarkupBuilder class know about our weather data use-case? Well, it didn't. But Groovy has meta-programming mechanisms that give developers a variety of facilities so that they can react to situations on the fly. As users of MarkupBuilder, we can feed it structures that meet our needs. MarkupBuilder will respond this way:

1. weatherData.rainfall starts off our XML structure and sets the root node to <rainfall>, so that everything within the closure argument is a child of that node.
2. The next node is a year element with a single property (value)—the associated closure sets the element's children.
3. Within a year element is a set of reading elements:

1. The reading has a number of properties that are set using map notation of key: value.
2. The content of each reading element is a rainfall measurement.

This makes for a very readable approach to creating the structure we need to represent in XML. For an XML node, we provide a name and (optionally) a set of map items for the element's properties. We can provide a value for the node once the properties have been provided. A closure is then started to add child elements.

Code Listing 45: MarkupBuilder for XML—Output

Because HTML is a close neighbor to XML, we can use MarkupBuilder to create (X)HTML. Code Listing 46: MarkupBuilder for HTML builds on the previous XML example to create an HTML file followed by the output provided in Code Listing 47: MarkupBuilder for HTML—Output.

The XML and HTML examples both use a rather static set of data, and we'll look into building the markup in a more dynamic manner shortly. As a brief aside, let me mention a number of items that might be worth further investigation, depending on your needs:

• If you want to generate HTML (or any other text format) using a template, Groovy's template engines are definitely worth a look. They're quite versatile, and they save using a third-party engine such as Velocity.
• Groovy provides a StreamingMarkupBuilder class that's useful for incrementally building up a structure.
• If you need to manipulate XML in place, check out the Groovy documentation.

Code Listing 46: MarkupBuilder for HTML

Tip: There's also a builder for JSON—the JsonBuilder.

Code Listing 47: MarkupBuilder for HTML—Output

Let's turn our MarkupBuilder skills toward the weather data. You'll see in Code Listing 49: Preparing XML Output that I've read in the data from the CSV and filtered the January 2010 readings. Calling new MarkupBuilder().weather, I'm preparing to convert those CSV-based records into XML (with <weather> as the root node). Yet again, I call on the each method to iterate through the data in order to create one <reading> element per CSV row. The format of the code may look a little odd at first, but remember that Groovy methods don't need parentheses, which means the body of the each closure is a single method call that the MarkupBuilder turns into an HTML element with properties. The top few elements can be seen in Code Listing 48: Sample Weather Output. 

Code Listing 48: Sample Weather Output

Code Listing 49: Preparing XML Output

A CSV-to-database example

Having worked through loading data from a CSV file and interacting with a database, we're now in a position to go a little deeper into a range of database tasks. In Code Listing 50: A Complete Database Usage Example, we'll construct a database from scratch, load it with CSV data, then run some queries. Instead of revisiting SQLite, we'll use an in-memory Apache Derby database.

Code Listing 50: A Complete Database Usage Example is a large script but can be broken down into the following sequential segments:

1. Load the required libraries, including the JDBC driver for Apache Derby ('org.apache.derby:derby:10.12.1.1').
2. Parse the CSV file with RFC4180.withHeader().parse(getFile('../../data/weather_data.csv').newReader()).
3. Setup an SQL instance: Sql.newInstance('jdbc:derby:memory:myDB;create=true'). Note the JDBC connection string includes a flag (create=true) that allows us to create the database if one doesn't exist. This may not be crucial for an in-memory database, but it would be important if we are using a persistent Derby database.
4. In order to create the required WeatherData table, we issue an sql.execute call with the appropriate data definition language (DDL) statement (CREATE TABLE).
5. The insertion of the CSV-based records into the WeatherData table is performed within a transaction using the withTransaction method. This is passed a closure that, when successfully completed, causes the transaction to be committed. Within the transaction, we make a series of parameterized INSERTs by calling sql.executeInsert.
6. Next, the first query is run in order to provide a summary of the data. We use sql.firstRow because we expect only one row to be returned.
7. The second query is for the rainfall readings for February 2012. Because this will return multiple rows, we use sql.eachRow, then pass it the query and a closure that will handle each row in the result set.
8. Just to be tidy, let’s then close the connection to the database with the call sql.close().

You might have noticed that I’ve used the Date class from the java.sql package when formatting the data for the insert operation. For each insertion, I call valueOf("$Year-$Month-$Day") class method from Date so as to bridge how Groovy deals with dates and how SQL deals with them.

Code Listing 50: A Complete Database Usage Example

Summary

This chapter has built on the basic language overview by demonstrating some very common programming processes. On nearly a daily basis, we are called on to read and write data from various sources and in various formats. I've presented the main formats you'll see (CSV, XML, JSON and databases) and also how far you can go by using Groovy's built-in feature set and existing libraries. This makes Groovy a very strong candidate for preparing nontrivial scripts for running on servers and desktops.