HDInsight and the Windows Azure Storage Blob

Loading Data into Azure Blob Storage

The HDInsight implementation of Hadoop can reference the Windows Azure Storage Blob (WASB) which provides a full-featured Hadoop Distributed File System (HDFS) over Azure Blob Storage.[6] This separates the data from compute nodes. This conflicts with the general Hadoop principle of moving the data to the compute in order to reduce network traffic, which is often a performance bottleneck. This bottleneck is avoided in WASB as it streams data from Azure Blob Storage over the fast Azure Flat Network Storage—otherwise known as the “Quantum 10” (Q10) network architecture)—which ensures high performance.[7]

This allows you to store data on cheap Azure Storage rather than maintaining it on the significantly more expensive HDInsight cluster’s compute nodes’ storage. It further allows for the relatively slow process of uploading data to precede launching your cluster and allows your output to persist after shutting down the cluster. This makes the compute component genuinely transitional and separates the costs associated with compute from those associated with storage.

Any Hadoop process can then reference data on WASB and, by default, HDInsight uses it for all storage including temporary files. The ability to use WASB applies to not just base Hadoop functions but extends to higher-level languages such as Pig and Hive.

Loading data into Azure Blob Storage can be carried out by a number of tools. Some of these are listed below:

A screenshot of CloudBerry Explorer connected to the Azure Storage being used in this example is below:

1. CloudBerry Explorer connected to Azure Storage

As you can see, it is presented very much like a file explorer and most of the functionality you would expect from such a utility is available.

Uploading significant volumes of data for processing can be a time-consuming process depending on available bandwidth, so it is recommended that you upload your data before you set up your cluster as these tasks can be performed independently. This stops you from paying for compute time while you wait for data to become available for processing.

Referencing Data in Azure Blob Storage

The approach to referencing data held in the WASB depends on the configuration of the HDInsight instance.

When creating the HDInsight cluster in the Management Portal using the Quick Create option, you specify an existing storage account. Creating the cluster will also cause a new container to be created in that account. Using Custom Create, you can specify the container within the storage account.

Normal Hadoop file references look like this:

HDInsight and PowerShell

PowerShell is the Windows scripting language that enables manipulation and automation of Windows environments.[9] It is an extremely powerful utility that allows for execution of tasks from clearing local event logs to deploying HDInsight clusters on Azure.

When HDInsight went into general availability, there was a strong emphasis on enabling submission of jobs of all types through PowerShell. One motivation behind this was to avoid some of the security risks associated with having Remote Desktop access to the head node (a feature now disabled by default when a cluster is built, though easily enabled through the portal). A second driver was to enable remote, automated execution of jobs and tasks. This gives great flexibility in allowing efficient use of resources. Say, for example, web logs from an Azure-hosted site are stored in Azure Blob Storage and, once a day, a job needs to be run to process that data. Using PowerShell from the client side, it would be possible to spin up a cluster, execute any MapReduce, Pig or Hive jobs, and store the output somewhere more permanent such as a SQL Azure database—and then shut the cluster back down.

To cover PowerShell would take a book in itself, so here we will carry out a simple overview. More details can be found on TechNet.[10]

PowerShell’s functionality is issued through cmdlets. These are commands that accept parameters to execute certain functionality.

For example, the following cmdlet lists the HDInsight clusters available in the specified subscription in the console: