Working with Data

---

This chapter presents some of the most common techniques that use data to create map features. Common data sources used with geo-applications include ordinary files stored locally, SQL data stored on a corporate network, and data stored remotely but accessible through a web service. Common data formats include plain text, Well Known Text (WKT), JSON, and GeoJSON.

In section 3.1, you’ll learn how to read data in WKT format from the Bing Spatial Data Service and how to filter and parse that data. You’ll also learn about color palettes and how to create choropleth maps. You’ll also learn about event-handling techniques and how to use object metadata.

Figure 16: Working with Data Demo

In section 3.2, you’ll learn how to read data in JSON from a web service and how to parse that data to create objects that can be added onto a map. You’ll also learn how to integrate JavaScript libraries, such as jQuery and AngularJS, into a Bing Maps geo-application.

In section 3.3, you’ll learn about color gradients and how to handle large data sets by creating heat maps. You’ll also learn how to access government geodata sources such as those provided by the U.S. Census Bureau.

3.1 Choropleth maps

A choropleth (pronounced koro-pleth) map uses different shades of a particular color to indicate the relative proportion of some statistic—for example, population or per-capita income. Creating choropleth maps using the Bing Maps V8 library is relatively simple. You define a color palette, get relevant shape and statistics data, apply a palette color, and add the result to a layer.

Figure 17: Choropleth Map Demo

The demo web application shown in Figure 17 displays a choropleth map of the United States by population. Darker shades indicate higher populations. The demo initially loads a map centered at (40.00, -96.00) and places a default-style purple pushpin at center. During initialization, a Layer object is created to hold the shapes of U.S. states.

When a user clicks the first button control, live shape and population data for the specified number of states is fetched using a REST service. A custom-color palette that consists of four shades of green is hard-coded. For each state, a color is determined based on the state's population, and the state’s polygon data is added to the display Layer object.

When the polygon for each state is drawn, a click-event handler is added that will display an alert dialog that has the target state’s name, population, and land area.

The demo web application is named ChoroplethDemo.html and is defined in a single file.

Code Listing 6: ChoroplethDemo.html

The demo application sets up three global script-scope objects:

var map = null;
var ppLayer = null;
var statesLayer = null;

Object ppLayer is a map layer for holding ordinary pushpins, in this case a single default-style pushpin to mark the map center. The statesLayer object is the main display layer for the choropleth coloring. The palette that defines the colors used by the choropleth is defined locally to the first button control’s click handler, but it could have been defined with the global objects.

The demo application loads the map object asynchronously by calling the program-defined GetMap() function in the usual way. The stateLayer object is instantiated in the GetMap() function.

All of the work is done inside the first button control’s click handler function. The function definition begins with:

function Button1_Click()
{
  var paletteGreen = ['rgba(203, 238, 131, 0.7)', 'rgba(138, 174, 67, 0.7)',
    'rgba(115, 150, 43, 0.7)', 'rgba(67, 102, 0, 0.7)'];

  WriteLn("Color palette = ");
  for (var i = 0; i < paletteGreen.length; ++i) {
    WriteLn(paletteGreen[i]);
  }
  var maxCount = textbox1.value;
. . .

There’s actually quite a bit of research on how many and which colors to use for a choropleth map, but I arbitrarily chose four shades of green and created a palette using the built-in rgba() function. You can also use color names. For example:

var paletteGreen = [ 'greenyellow', 'lightgreen',
  'mediumseagreen', 'darkolivegreen' ];

The maximum number of states to process (20 in the demo) is pulled from the textbox1 control. It’s not necessary to cast the value to an integer in this situation.

There are several ways to get choropleth data. The demo application uses the Bing Maps V8 SpatialDataService module to get live state data from a Bing Maps RESTful web service. All of the work is done inside the module’s callback function:

Microsoft.Maps.loadModule('Microsoft.Maps.SpatialDataService', function () {
  var world = Microsoft.Maps.LocationRect.fromEdges(90, -180, -90, 180);
  var queryOptions = {
    queryUrl: 'https://spatial.virtualearth.net/REST/v1/data/' +
      '755aa60032b24cb1bfb54e8a6d59c229/USCensus2010_States/States',
    spatialFilter: {
      spatialFilterType: 'intersects',
      intersects: world
    },
    top: maxCount
  };

There are several different types of queries supported by the web service, but most require some sort of spatial filter to narrow the result list down. For example, if you query the service for points of interest, you’d probably want to limit your results to a city or a region. The demo sets its filter to the entire world because there will only be a maximum of 52 results returned (the 50 states plus Puerto Rico and the District of Columbia).

Next, the search() function is invoked:

Microsoft.Maps.SpatialDataService.QueryAPIManager.search(queryOptions,
  map, function (data) {
  WriteLn("\nData from SpatialDataService: \n");
  for (var i = 0; i < data.length; ++i) {
    var name = data[i].metadata.Name;
    var pop = data[i].metadata.Population;
    if (name == "Alaska" || name == "Hawaii") {
      continue;
    }
    WriteLn(name + " " + pop);
    data[i].setOptions({
      fillColor: GetStateColor(pop, paletteGreen)
    });
    Microsoft.Maps.Events.addHandler(data[i], 'click', function (e) {
      var n = e.target.metadata.Name;
      var p = e.target.metadata.Population;
      var a = e.target.metadata.Area_Land;
      alert(n + '\nPopulation = ' + p + '\nArea = ' + a);
    });
  }

The callback function returns results into an object named data. Each item in data has a metadata property that in turn has a Name, Population, Land_Area, and other information. I determined the metadata names during development by placing this statement in the for-loop:

WriteLn(Object.keys(data[i].metadata));

The demo performs a secondary programmatic filter to remove Alaska and Hawaii. The color for each state is set by calling a program-defined GetStateColor() function. Then each data item has an event handler added that will display metadata information in a humble JavaScript alert box. Using a custom Infobox object is an alternative.

The function that determines color value from population is:

function GetStateColor(pop, palette)
{
  if (pop < 2000000) {
    return palette[0];
  }
  else if (pop < 5000000) {
    return palette[1];
  }
  else if (pop < 8000000) {
    return palette[2];
  }
  else {
    return palette[3];
  }
}

The function is simple but effective because only four colors are being dealt with. When you have many different choropleth colors, one alternative approach is to define an array of threshold values. For example:

var popLimits = [ 0, 2000000, 5000000, 8000000 ];

You could write code that accepts a population value, scans through the threshold values, and returns an index into the palette array.

In summary, it’s relatively easy to create a choropleth map once you have shape data and associated statistics data. The most difficult part is getting shape data. You can use the virtualearth.net REST service, one of many external data sources available, and you can parse the results of that service by using the search() function in the Bing Maps V8 library SpatialDataService module. Alternative approaches for creating a choropleth map include getting data as plain text, WKT, GeoJSON, or standard JSON format from a government or commercial web service and parsing the data programmatically.

Resources

For detailed information about the Bing Maps V8 Spatial Data Service module, see:
https://msdn.microsoft.com/en-us/library/mt712849.aspx.

For detailed information about the Query API component, see:
https://msdn.microsoft.com/en-us/library/gg585126.aspx.

You can see collections of shades for various colors at:
http://www.w3schools.com/colors/colors_groups.asp.

3.2 Data from a web service

One of the most common ways to obtain data for a map application is to query a web service. In most cases a web service returns data in JSON or XML format. An effective technique for getting data for a Bing Maps application from a web service is to use the jQuery library. Because the Bing Maps library is simply ordinary JavaScript, using the jQuery library is a natural approach.

Figure 18: Data from a Web Service Demo

The demo web application shown in Figure 18 gives an example of fetching JSON data from a simulated web service. The demo initially loads a map centered near Morrisville, N.C., and places a default-style large purple pushpin at the map center.

The text box control in the upper left points to a data file named LatLonData.json on the local web server, but the technique presented in this section can use any URL that points to a web service that returns JSON data. When the user clicks “Get Pushpins,” a request is sent to the simulated web service. The returned data is echoed and used to place four pushpins on the map. Each pushpin has a mouse click event handler that displays an Infobox object containing data about the associated pushpin.

The demo application is named DataFromWebServiceDemo.html and is defined in a single file.

Code Listing 7: DataFromWebServiceDemo.html

A reference to the jQuery library is placed at the top of the HTML head section:

<script type="text/javascript"
  src="http://code.jquery.com/jquery-latest.min.js"></script>
<script type='text/javascript'>

The idea is to use the jQuery library to fetch JSON data from a web service. An alternative is to use the built-in JavaScript XMLHttpRequest() function. In general, I try to avoid external dependencies, but the jQuery library’s ajax() function is very convenient.

Most of the work is performed by the event handler for the HTML button1 control. The structure of the handler is:

$(document).ready(function() {
  $("#button1").click(function() {
    var urlVal = $("#textbox1").val();
    WriteLn("Fetching data from: \n");
    WriteLn(urlVal + "\n");

    var radius = parseInt((textbox2.value).split('=')[1]);
    var pinkDot = CreateSvgDot(radius, "DeepPink");       

    $.ajax({
      url: urlVal,
      dataType: "text",
      success: function (data) {
        WriteLn("Retrieved data is: \n");
        WriteLn(data);
        var parsed = JSON.parse(data);
        var len = parsed.d.length;
        for (var i = 0; i < len; ++i) {
          // make pushpin, add to global pushpins[] array
        }

        ppLayer.add(pushpins);
        map.layers.insert(ppLayer);
      }
    });
  });
});

The code begins by grabbing the URL from the textbox1 control using jQuery syntax with the val() function, then echoing the URL to the HTML message area. In a production scenario, at this point you’d probably want to do some error checking of the URL value.

Next, a pink dot, to be used as the pushpin icon, is created dynamically by a call to a program-defined CreateSvgDot() function that I will explain shortly. First, the desired radius is fetched from the textbox2 control. I used standard non-jQuery syntax in order to show the difference.

The ajax() function can accept 34 settings, but the default values for most of the settings are used. The key setting in the demo is the success setting, which is a callback function that will execute after the data has been returned asynchronously.

The JSON data returned from the simulated web service is:

{ "d" :
  [
   { "lat":35.76, "lon":-79.20, "desc":"top-most data" },
   { "lat":35.68, "lon":-79.20, "desc":"bottom-most data" },
   { "lat":35.72, "lon":-79.24, "desc":"left-most data" },
   { "lat":35.72, "lon":-79.16, "desc":"right-most data" }
  ]
} 

The top-level field is tersely named “d,” and the contents consist of an array of four items in which each item has a latitude, a longitude, and a description. The JSON.parse() function is used to break the data into key-value pairs, and the result is stored into a local object named parsed.

The code that creates the pushpins is:

for (var i = 0; i < len; ++i) {
  var loc = new Microsoft.Maps.Location(parsed.d[i].lat,
    parsed.d[i].lon);

  var ppOptions = { icon: pinkDot,
    anchor: new Microsoft.Maps.Point(radius,radius) };

  var pp = new Microsoft.Maps.Pushpin(loc, ppOptions);
  pp.metadata = parsed.d[i].desc;
  pushpins[i] = pp;
  Microsoft.Maps.Events.addHandler(pp, 'click', ShowInfobox);
}

The description information is added to each pushpin object as a property named metadata. The code here is using the rather strange JavaScript ability to add arbitrary properties to objects, which means the name metadata is not required and the code could have been written as:

pp.foo = parsed.d[i].desc;

Each pushpin has its click event modified so that control will be transferred to a program-defined function ShowInfobox(). Alternatively, you can directly define the actions you’ll take by using an anonymous function.

The demo creates custom pushpin icons using function CreateSvgDot(), which is defined this way:

function CreateSvgDot(radius, clr)
{
  var s = '<svg xmlns="http://www.w3.org/2000/svg"';
  s += ' width="' + radius * 2 +'"';
  s += ' height="' + radius * 2 + '"';

  s += '>';
  s += '<circle cx="' + radius + '"';
  s += ' cy="' + radius + '"';
  s += ' r="' + radius + '"';
  s += ' fill="' + clr + '"';
  s += ' /></svg>';
  return s;
}

An example of an SVG image is:

<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16">
  <circle cx="8" cy="8" r="8" fill="red" />
</svg>

So, one technique for creating an SVG image on the fly is to write a function that returns a big string. An alternative is to use the built-in JavaScript createElementNS() and setAttributeNS() functions.

The ShowInfobox() callback function is executed when a user clicks a pushpin:

function ShowInfobox(e)
{
  var loc = e.target.getLocation();
  WriteLn('\nmouse click at ' + LatLonStr(loc));
  infobox.setLocation(loc);
  infobox.setOptions({
    visible: true,  title: e.target.metadata, description: LatLonStr(loc)
  });
}

Notice that the location of the Infobox is determined by programmatically fetching the location of a pushpin. Alternatively, when pushpin locations do not change, we can store pushpin locations in an array when they are created and return an index into the array when the mouseover or click event is fired.

In summary, when you need external data for a geolocation map application and that data is exposed through a web service that returns JSON or XML data, using the ajax() function in the jQuery library is a good approach. You can create a dynamic, custom icon image for a pushpin object by writing a function that returns an SVG string.

Resources

For detailed information about SVG images, see:
https://developer.mozilla.org/en-US/docs/Web/SVG/Element.

For detailed information about the jQuery ajax() function, see:
http://api.jquery.com/jquery.ajax/.

3.3 Heat maps

When dealing with large numbers of latitude-longitude data points, an alternative to displaying the data points as separate pushpins (or as clustered pushpins) is to display the data as a heat map. There are several kinds of heat maps, and one common type displays combined data points using a color gradient where different colors represent different data densities.

Figure 19: Heat Map Demo

The demo web application shown in Figure 19 initially loads a map centered at (37.50, -118.00) and places a default, large purple pushpin at center. First, the user clicked “HTML5 File Browse” and pointed to a local file named NV_Cities.txt that contained city data. Next, the user clicked the first button control, which loaded and displayed a heat map for city density in the state of Nevada. The user then cleared that heat map by using the second button control.

Note that the user next clicked “Browse” and pointed to a tab-separated text file named CA_Cities.txt. That data file contains a list of 1,522 cities in California and the corresponding latitude-longitude information for each. Next, the user clicked “Show Heat Map,” which read the text file, parsed out the lat-lon data, and stored that data into an array. Then the lat-lon data was displayed as a heat map, generating a city-density visualization.

The demo web application is named HeatMapDemo.html and is defined in a single file.

Code Listing 8: HeatMapDemo.html

The demo application sets up seven global script-scope objects:

var map = null;
var ppLayer = null;
var hmLayer = null;
var reader = null;
var locs = [];
var cGrad = { '0.0': 'black', '0.2': 'purple', '0.4': 'blue',
  '0.6': 'green', '0.8': 'yellow', '0.9': 'orange',
  '1.0': 'rgb(255,0,0)' };
var hmOptions = { intensity: 0.65, radius: 7, colorGradient: cGrad };

Object ppLayer is a map layer for holding ordinary pushpins, in this case a single pushpin that marks the map center. Object hmLayer is a map layer for holding the single heat map. Object reader is an HTML FileReader object to read() the source data. Array object locs holds all the lat-lon data from the source data file.

Object cGrad is a generic JavaScript key-value object that defines the color gradient for a heat map. The keys are values between 0.0 and 1.0 that control heat map colors. The values are colors that can be defined using strings or using the rgb() or rgba() functions.

Object hmOptions defines the heat map options. The intensity key can take a value between 0.0 and 1.0 in which larger values produce a brighter heat map. If omitted, the default value is 0.5. The radius key defines the size of each data point that defines the heat map, and the default value is 10 pixels. Larger values produce a heat map with fewer gradations.

As with the demo, the colorGradient key accepts a program-defined color gradient. The colorGradient is optional and, if omitted, the default values used are:

{
  '0.00': 'rgb(255,0,255)', // Magenta
  '0.25': 'rgb(0,0,255)',   // Blue
  '0.50': 'rgb(0,255,0)',   // Green
  '0.75': 'rgb(255,255,0)', // Yellow
  '1.00': 'rgb(255,0,0)'    // Red
}

There is also an optional opacity key, which is not used by the demo. The default value is 1.0, which will create no-see-through colors—values such as 0.5 allow you to see labeling under the heat map. There is an optional unit key that defines the units used for values of the radius key. The default is the string value “pixels.” When the map area is very large (entire earth), the single alternative is “meters.”

Table 3: The HeatMapLayerOptions Object

The demo application loads the map object asynchronously by calling the program-defined GetMap() function in the usual way. If you want to display the colorGradient object at load time, you can do so with code similar to this:

for (key in cGrad) {
  WriteLn(key + " " + cGrad[key]);
}

The button control labeled “Show Heat Map” has ID property “button1” and the onclick attribute points to function Button1_Click(), which is defined as:

function Button1_Click()
{
  if (reader == null || locs.length == 0) {
    LoadLocs();
  }
}

So, function Button1_Click() is merely a wrapper to a call to a program-defined LoadLocs() function. There’s no technical reason why the code in LoadLocs() can’t be placed directly inside function Button1_Click(), but the two-function approach gives you a bit more flexibility.

The checks to the reader object and the locs.length property will become clear when we address function Button2_Click(). Note that in many situations, you'd want to check the length property as:

if (reader == null || locs == null || locs.length == 0) {

The application logic, however, ensures that the locs array will never be null. Function LoadLocs() does most of the work. The function code is:

function LoadLocs()
{
  var f = file1.files[0];
  WriteLn('Loading data from ' + f.name + "\n");
  reader = new FileReader();
  reader.onload = function(e) {
    // parse each line, create Location, store into locs

    Microsoft.Maps.loadModule('Microsoft.Maps.HeatMap', function () {
      hmLayer = new Microsoft.Maps.HeatMapLayer(locs, hmOptions);
      map.layers.insert(hmLayer);
    });
  }
  reader.readAsText(f);
}

Here’s the key idea—when the FileReader onload event fires, control is transferred to the anonymous callback function that parses each line of the data file and populates the locs array. The locs array is fed to the HeatMap callback, creating the hmLayer object that is then inserted onto the map.

The line-parsing code is:

var lines = reader.result.split('\n');
for (var i = 0; i < lines.length; ++i) {
  var line = lines[i];
  var tokens = line.split('\t');
  try {
    var loc = new Microsoft.Maps.Location(tokens[12], tokens[13]);
    locs[i] = loc;
  }
  catch (err)  {
    WriteLn(err + " " + line);
  }
}

A lot can go wrong when we parse a text file, so the demo uses a try-catch when constructing each Location object. The source data files look like this:

CA  602000  2409704  Anaheim city    ( . . )  33.855497  -117.760071
CA  602028  2628706  Anchor Bay CDP  ( . . )  38.812653  -123.570267
. . .

Each line has 14 tab-delimited values. The first value is the state abbreviation. The next two fields are IDs. The fourth field is the place name, which can be a city, a town, or a Census Designated Place (CDP). Next are eight other fields, including U.S. census population count and land area. The last two fields are the latitude and longitude. You can get files CA_Cities.txt and NV_Cities.txt from https://github.com/jdmccaffrey/bing-maps-v8-succinctly.

The button control labeled “Clear Heat Map” is associated with function Button2_Click(), which is defined:

function Button2_Click()
{
  WriteLn('Clearing heat map' + "\n");
  hmLayer.clear();
  reader = null;
  locs = [];
}

The clear() method of the HeatMapLayer class deletes a heat map layer. Useful alternatives are the hide() and show() methods.

The FileReader object is set to null in order to force a file read the next time the LoadLocs() function is called. And the locs array is reset to an empty array so that the next file read doesn’t append to the existing locations.

In summary, in order to create a heat map, you use the HeatMapLayer object in the Microsoft.Maps.HeatMap module. The HeatMapLayer object requires an array of Location objects and, optionally, a HeatMapayerOptions object to customize the color gradient and other visualization characteristics.

Resources

For detailed information about the HeatMapLayer class, see:
https://msdn.microsoft.com/en-us/library/mt712811.aspx.

For detailed information about the HeatMapLayerOptions class, see:
https://msdn.microsoft.com/en-us/library/mt712810.aspx.

You can see some example custom-color gradients at:
https://msdn.microsoft.com/en-us/library/mt712854.aspx.

The source data files in .zip format can be found at the U.S. Census website at:
https://www.census.gov/geo/maps-data/data/gazetteer2010.html.

You can find interesting map data at the U.S. Geological Survey website at:
http://nationalmap.gov/.