Web map based data visualization

1. Pseudo-Animation Move on road for direction on Google Maps First you select a source-destination, then the app will use Google Maps JS API to get a route, and update the bus marker on the map continuously. It’s not a real animation, but plotting the marker frequently.  direction
2. Animation on Google Maps + Paper.jsThis the combination of Paper.js and Google Maps JS API. It demonstrates the possibility of rendering self-contained canvas on top of Google Maps canvas.paper
3. Leaflet MarkerClusterGood Markercluster plugin for Leaflet, especially for the nice animation.cluster
4. Mapbox Courier Example for Logisticscourier
5. Taxis in NYC Amazing data visualization of a daily life of taxi driver in NYC. The creator described the whole tech in the blog. Some one introduces a brief skill on this. Cursor_and_A_Day_in_the_Life
6. Digital Attack Map
Showing live DDOS attack across the globe, from Google Ideas.
7. Events Visualization

A bug in ArcGIS REST Service

We run ArcGIS servers in two different environments DEVELOPMENT and UAT , and the guy who set up the infrastructure has gone.
Last week we deployed the working app into the UAT, and it didn’t work as supposed as in DEVELOPMENT.

Specifically,  we had problem with the ArcGIS login process. We use ArcGIS REST API token generation for login authentication, and the app host address as HTTP referer.
Details see http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/Generate_Token/02r3000000m5000000/.
After user logs in, we use the generate Sitemap in REST service to get a list of access-controlled layers for the user.
We don’t use the REST method as it only returns the current folders and then again requires ajax under the folder to get all the layers, it slows down the whole app.

In DEV, we successfully get the sitemap XML with all the layers as shown like this.

However, we could not manage it in the UAT, and it always returned an empty XML, i.e. no layer services at all.

Token returned from UAT server was correct, and there was nothing wrong with code and configuration. We debug the login module, the token generation process and any possible out-dated codes. Nothing worked.

The bug cost me one week to catch.  See Server 1 as DEV and Server 2 as UAT, something is missing:


I doubt that the guy who implemented the REST interface was different from the one who implemented the Sitemap interface. The later just ignored the whole folders when there was None Services, but this None only means there is no service under the root. Damn!

Bypass IdentityManager Popup in ESRI ArcGIS JavaScript API

When using ArcGIS secure layers in JavaScript API 2.5+, the default IdentityManager will come out as a login process for web app users.Please see example at http://developers.arcgis.com/javascript/samples/widget_identitymanager/ What if I already have a login for a web app, and do I need to login again for the map layers? What can be done to avoid the IdentityManager Popup ? Login use ArcGIS REST API: see http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#//02r3000000m5000000 And then bypass the IdentityManager Popup.

//just use AJAX, you can put this login in any page
var timeout = 1440, referer = hostname + '/arcgis/rest';
var postData = {
    'username': username,
    'password': password,
    //'client': 'requestip', //can't work if using this type of request
    'referer': referer,//should be the web app host name
    'expiration': timeout,
    'f': 'json'//,
    //'encrypted': 'true' //if need to enable, please add the necessary js files, see generateToken page source code from ArcGIS
//processing data encryption
//postData = encryptFormData(postData, modulus);
   url: url,//hostname/arcgis/tokens/generateToken
   cache: false,
   type: 'POST',
   dataType: 'jsonp',
   data: postData,
   crossDomain: true,
   success: function (response) {
    if (response) {
        if (response.error) { // access error
         return false;
        } else {
         //store username and token in cookies
         setCookieName("arcgisusername", username, 12);
         setCookieName("arcgistoken", response.token, 12);
     } else {
        return false;
   error: function (error) { // system error
    return false; 

//bypass the popup
//require IdentityManager
//now build a identitymanager json object to init the object
var now = +(new Date());
var expires = now + (timeout*60000);
var imObject = {
"serverInfos": [
"server": hostname,
"tokenServiceUrl": hostname + "/arcgis/tokens/",
"adminTokenServiceUrl": hostname + "/arcgis/admin/generateToken",
"shortLivedTokenValidity": timeout,
"currentVersion": 10.22,//update necessary
"hasServer": true
"oAuthInfos": [],
"credentials": [
"userId": getCookieName('arcgisusername'),
"server": hostname,
"token": getCookieName('arcgistoken'),
"expires": expires,
"validity": timeout,
"ssl": false,
"creationTime": now,
"scope": "server",
"resources": [
hostname + '/arcgis/rest/services'

//Please remember that if you need to query some layer data using ajax or using some layer URL,
// you need to manually append the '&token=sometokenfromserver' for the URL

I found another hack to bypass the ArcGIS IdentityManager popup or a new method to access secure layers.

This requires coding in .NET/C#. I figure out that ArcGIS server use request cookie name agstoken as an authentication method for accessing secure layers.So if I can create a login form and post to the Login.aspx and send the response with cookie contains agstoken which I can freely request from ArcGIS services using the REST API.

using System;
using System.Collections.Generic;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Net;
using System.IO;
using System.Text;
namespace Login
    public partial class _Login : System.Web.UI.Page
        protected void Page_Load(object sender, EventArgs e)
            //code for normal user login, if failed, then redirect to login page
  	    string h = "hostname";
	    //h = Request.UserHostAddress;//may not work as direct ip address
            Response.AddHeader("Access-Control-Allow-Origin","*"); //may tighten the access
            Response.AddHeader("Access-Control-Allow-Methods", "POST");
            Response.AddHeader("Access-Control-Allow-Headers", "Content-Type, Accept");
	    HttpCookie c2 = Request.Cookies.Get("agstoken");
	    HttpCookie c3 = Request.Cookies.Get("agsapperror");	
	    string timeout = "1440";//mins
	    string u = Request["u"].ToString();
	    string p = Request["p"].ToString();
	    string referer = h;
	    string tokenURL = h + "/arcgis/tokens/generateToken";
            string type = "application/x-www-form-urlencoded";
            string token = "";
            WebClient wc1 = new WebClient();	
	          string loginData = "username="+u+ "&password="+p+     
                  wc1.Headers[HttpRequestHeader.ContentType] = type;
	          string loginContent = wc1.UploadString(tokenURL, loginData );
	          //check result      
			if(c3 == null){
			     c3 = new HttpCookie("agsapperror");
			     c3.Value  = "userpasswordwrong";
			     c3.Expires = DateTime.Now.AddMinutes(10d);
		          //push user token for accessing secure layers
			  token = getUserToken(loginContent);
                          if(c2 == null){
				c2 = new HttpCookie("agstoken");
				c2.Value  = token;
				c2.Expires = DateTime.Now.AddMinutes(1440d);
	       } catch (WebException e1) {
                   //Response.Write("{\"error\":\"network connection error\"}");
	public string getUserToken(string tokens){
	    string token = ""; 
               string [] r = tokens.Split(new string[]{"\",\""},StringSplitOptions.None);
	       if(r.Length > 0){
                   string part1 = r[0];
                   string [] r2 = part1.Split(new string[]{"\":\""},StringSplitOptions.None);
                   token = r2[1];   
	  return token;

Maps/GIS in Singapore Summary

I have few years of software development of various maps API, and I want to make a summary of my experience.

For simple website embedded maps, the most used is Google Maps, and it is the easiest one to implement. Users just need to tell a location in a map, and that is it.

For business usage like foursquare, streetdirectory, they are using customized OpenStreet maps, and 3rd party map API like OpenLayers, Mapbox, and Leaflet.

For government related applications, the most used is ESRI ArcGIS maps, which is so-far the most powerful player in enterprise GIS industry. ESRI ArcGIS is mostly for professionals, and they have fully packed GIS tools and  APIs targeting different platforms.

There is a rising trend using Tableau for Data Analytics interaction with maps, although they have really simple and slow maps.

Singapore Online Maps for normal users:

  1. Google Maps: easy to use and implement
  2. Gothere.sg: super fast and up to date
  3. Here.com/Bing Maps: minority usage
  4. Esri Maps: rare usage 

Government/Organization Maps:

  1. OneMap by SLA: flash based, but lots of accurate data up to date, the only one provides public JavaScript API in Singapore content
  2. Whereto.sg by ESRI Singapore: flash based, average map data
  3. URA Maps by URA: flash based, master plans, car park, private housing
  4. HDB Maps by HDB: simple but good for HDB data

Development Experience:

    1. Google Maps
      1. Google Maps JavaScript API V3
      2. ArcGIS Link for Google Maps JS V3: We use this link for consuming ArcGIS service and simple query tasks.
      3. For Google Maps, they also have Geocoding Service API, Places API, Static Map API and the Street View Image API, etc.
      4. If you have a good sense of design and lots of time, you can style Google Maps  online:  https://snazzymaps.com/explore, and it is very simple to display a styled map using JS API.
         var mapStyle = [....];
         var style = new google.maps.StyledMapType(mapStyle, { name: styleName });
         map.mapTypes.set(styleName, style); 
  1. Here/Bing Maps: no experience
  2. Esri Maps: a lot of fun..
    1. a. ArcGIS JavaScript API: most popular
    2. b. Native mobile APIs: not easy to do
    3. c. Flash/Sliverlight APIs: no longer popular as HTML5 rising
    4. d. .NET/C#, Java APIs for desktop application
    5. e. REST API: if you want to use a lot of AJAX, remember to set dataType to JSONP for IE….
  3. OneMap
    1. a. http://www.onemap.sg/api/help/, not really a help but all ugly documented, basically JavaScript
    2. b. Before you need to register http://www.onemap.sg/home/apireg.aspx
    3. c. They have lots of public data for displaying on map, which is avaiable through data.gov.sg. For instance, schools, hospitals, parks, sports centers, etc.

Spatial References: It is definitely a headache to parse points in different coordinates system.

  1. 4326 (WGS 1984 ) in decimal degrees: Google Map, common GPS data format
  2. 102100/3857 in meters: WGS84 Web Mercator
  3. 3414/SVY21 in meters Singapore SLA
  4. 4757 SVY21 in decimal degrees Singapore

For ArcGIS, there is a GeometryService which can convert coordinates in different projection, and you can use AJAX to accomplish this without using ESRI JS API.


TODO task for learning ASP.NET MVC 5:

1. Consume ArcGIS REST API service in server side, i.e. generate token service for login

2. Customize login page and redirect, aka. routing issue

3. User registration module for application with SQL Express

4. Port ePlanner 2.0 to MVC and make it run

5. Write more using C#

Finish the book Professional ASP.NET MVC 5.


1. Make Yahoo! Web Service Rest API Call using C#

2. Using Secured Web Services and Tokens

string url = tokenServiceUrl + "?request=getToken&username=
myuser&password=secret"; System.Net.WebRequest request = System.Net.WebRequest.Create(url);

string myToken, errorMsg;
try {
   System.Net.WebResponse response = request.GetResponse();
   System.IO.Stream responseStream = response.GetResponseStream();
   System.IO.StreamReader readStream = new System.IO.StreamReader(responseStream);

   myToken = readStream.ReadToEnd();
catch (WebException we) {
   if (we.Message.Contains("403")
      errorMsg = "Server returned forbidden (403) code."

3. How to Customize ASP.NET MVC Authentication

4.ASP.NET MVC Authentication – Customizing Authentication and Authorization The Right Way

5.Sesseion Hijacking

6. Read Web.config data
in Web.config add:

<add key="customsetting1" value="Some text here"/>


			System.Configuration.Configuration rootWebConfig1 =
			if (rootWebConfig1.AppSettings.Settings.Count > 0)
				System.Configuration.KeyValueConfigurationElement customSetting = 
				if (customSetting != null)
					Console.WriteLine("customsetting1 application string = \"{0}\"", 
					Console.WriteLine("No customsetting1 application string");

Map Cluster Layer Heuristics Algorithm

I am implementing an Cluster Layer with heuristics algorithm  based on Google Maps JS API with reference to MarkerCluster.

It works like this:

1. Set a maximum zoom level Max

2. If map.getZoom() > Max, remove the cluster layer, and display the original point layer

3. If map.getZoom() <= Max, remove the original point layer

If init load on minimal zoom level, query all points out, and store on client side, later use this data to Filter cluster layer markers based on map.getBounds(), i.e. map extent

If map load on between minimal zoom level and Max, query data based on map.getBounds()

4. Filter function

a. get xmin, ymin, xmax, ymax from the map bounds.

b. filter points within the map bounds to redraw Cluster Layer