Today’s web user can create content on the web both collaboratively and individually, allowing for a personalised web experience through wikis, blogs, podcasts, photo sharing, and other technologies. GIS and mapping applications have both benefited from and contributed to these trends, collectively called “Web 2.0”. Various applications are available for taking part in the new cartography: geotags, geoblogs, geowikis, GeoRSS feeds, Google MyMaps, and Yahoo Pipes. These spatially aware methods and applications require little or no programming expertise, and show that with the modern web, it”s all about location, location, location. And for those that know a little bit of web scripting, map mash-ups with mapping tool APIs have considerable power to integrate data from multiple sources onto an online map.
Background to experimentation
Mashups and related technologies are revolutionising the way users can build interesting and useful applications over web. It is otherwise a difficult task in a conventional application development scenario. It is a concept, rather than a newly evolved programming paradigm, that was indeed being used in conventional application development in an unknowing way and lack of standards for data and application interoperability was felt a bottleneck for its wide spread usage. The popularity of mashups today is to be attributed to the wide availability of data and services / APIs , in the recent years, coupled with ubiquitous availability that in a way a casual user today can begin programming the Internet to build interesting and useful applications with creativity and innovation.
In this paper we provide insights specifically into client-side map mashup development based on our practical experiences in implementing various sample applications and tools for creating them by examining current development of mashup technologies. The paper also highlights the current status of major map content providers in India, their web services and API availability for enabling useful end-user map mashups. The main emphasis of research and experimentation work carried out was to address the following challenges like:
- To quickly build an online mapping application with user provided data
- To extract content from the current static websites that do not offer data as standard feeds
- To aggregate and filter out contents from multiple feed data
- To extract geo-location information from content that explicitly does not provide coordinate information
- To visualise data temporally
- To identify client-side development environments that exist specifically in the form of visual programming environments that help non-programmers
- To host new services generated out of mashing up
- To push the output results using e-mail services or instant messaging services as alerts
To address the above, experimentation and research work was performed aimed at assessing the possibility of achieving the intended task; the effort involved in programming or required coding capabilities, whether the required data sources are available over public domain and the formats or the presentation of data, and the usability of the implementation as a client-side mashup. Also, the authors would like to mention that the idea behind carrying out such work was purely research oriented and targeted towards understanding new technological developments and capabilities in using the Internet as a programming platform.
Task -1: To add GIS layers on top of existing online base map data
The objective of attempting this task was to understand how a base map from a publicly available data provider can be used with regard to customizing or programming it as per requirements at the client-side. In this regard a survey of map data providers was done to identify suitable sources. There exist several data providers that render geo-spatial data depicting the entire world in an extremely fast rendering on-line application within a web browser. Prominent among them are map data providers like Google maps, Yahoo maps, Microsoft Bing maps, India’s Bhuvan , ESRI ArcGIS online, openstreet maps to name a few that represent the required background map data. In the context of adding user specified GIS map data or layers upon above mentioned base maps, only a few public cloud based offerings of building map content were found to be suitable, from a non-programmers point of view, like GIScloud.com , ArcGIS.com , Mapbuilder.net , GeoCommons.com etc. that meet requirements of
- creation a spatial reference frame-work and study-area
- layer based viewing with visibility on/off setting
- layer style setting (symbology), setting transparencies
- catalog search for required layers for making a base map
- generation of online web applications with custom tools to operate on map
- customizing through scripting etc.
One of the public cloud based services, ArcGIS.com, was chosen to implement the task of adding and displaying layers of India states and waterline theme. It was found to be more intuitive in using the frame-work. Simple steps were followed in setting the spatial reference system, adjusting the back ground layer to the required extents and then uploading the layers as shape files to the service. The service quickly added them as layers on top and followed with steps for adjusting the layer style settings. Finally the map could be published either through making it available for public viewing, create an online web application or a through a link that could be used for hosting on user’s own web server as shown in fig-1.
Fig-1 Data layers added to yahoo maps background
Task-2: Creating live RSS feeds of content extracted from static websites
The objective of attempting this task was to create useful feeds as standard RSS /JSON or other formats from static websites. In this regard the Website of Jawaharlal Nehru Technological University, Hyderabad ) was chosen to extract required information and generate a live feed. The website puts up information on a web notice board for its students to know the details regarding notifications, examination schedules and other required information as shown in fig-2.
Fig-2 yahoo pipe implementation of web scraping
The information as such is presented in HTML form and a student is expected to visit the web site often to get to know about an update. The idea behind creating a live feed is to benefit a student by giving an alert to his or her e-mail or mobile phone as the site gets updated. Good web scraping techniques and tools exist for such kind of implementations. After, some basic experimentation with a few tools, the authors found Yahoo pipes to be the most useful in this regard. Though another tool named Dapper  was found to be more powerful with its interactive scraping techniques, at the time of this writing, it was known to be acquired by yahoo. Yahoo pipes provided useful building blocks for scraping, filtering out extra characters and creating the live feed. The visual programming through piping was done to scrape only the web notice board content from the website organize the content as bulletin board, notifications and R&D downloads. The resulting live RSS feed was burnt with Google’s feed burner (https://feeds.feedburner.com/RSSfeedOfJntuhWebsite) so that it is publicly available and the live feed accessed is as shown in fig-3. Students or users subscribed to the feed get the benefit of getting the required update as an alert to their e-mail or mobile phones.
Fig-3 RSS feed generated of JNTUH website
Task – 3: Feed aggregation and filtering based on a user entered keyword
The objective of attempting this task was to aggregate various related RSS feed content, filtering the consolidated feed content with user entered keyword and extracting geo-location information.
In this task four news related RSS feeds from different sources namely The Hindu, NDTV, Rediff and Frontline were first aggregated and passed through a filter for extraction of all the titles of feeds with user entered keyword. The implementation of the task was attempted in yahoo pipes. The result of a filtered output for a user entered keyword “lokpal” is shown in figures 4. Such a task performs a useful job for a user to be alerted on specific topic or keyword of interest from aggregated news feed.
Fig-4 results of feed aggregation and filtering with user entered keyword
Task – 4: Creating a map mashup by Geo- tagging feed information and displaying it in a spatio-temporal fashion
The objective of attempting this task was to build an interesting map mashup by geo-tagging RSS feeds of news and overlay the contents both on a map and on a time-line.
This task is an attempt to build a client-side map mashup with RSS feed as input source, geo-names API to act on input data and Yahoo maps API to finally display the output content. The Hindu news, as of this writing, was not available as a GeoRSS feed so that useful geo-location information may be directly derived. Hence, for this task, the first step of implementation was to use Geo-names API for converting The Hindu news RSS feed to GeoRSS feed. It is important to note that the Geo-names API ‘RSStoGeoRSS’ convertor method performs a check with each feed item’s title and description for a known place name and returns its geographic coordinates (latitudes and longitudes). Next, the resulting GeoRSS feed was parsed for extracting the location information to be used with yahoo maps. Also, an attempt was made to display the above feed over a time-line. For this purpose “timeglider” application was chosen. This application was found to be useful as it allows direct import of feed contents. The time-stamp information from the feed was automatically extracted and the results are shown in figure-5.
Fig – 5 spatial and temporal visualization of news feed from “The Hindu”
Scenario in India
In India, huge amount of geo-spatial data has been prepared over years that are available largely with Government bodies. Several initiatives are being taken by Government of India and state governments, as part of open data policy, to provide these data sets existing in either geo-spatial form of maps (topographic and thematic), satellite based imagery or associated data like Climate and weather, Census and demographic data, socio-economic statistical data, other financial and market related data, as services for users to access them as feed data and API’s to build useful client-side mashups. Notable in this regard are Indian Space Research Organization (ISRO) initiative of Bhuvan (delivering 3D Virtual Earth experience and 2D map display of satellite imagery and thematic maps) that offer API for client side programming and other services, Indian Meteorological Department (IMD) offers data of earth quake information, through its website, as RSS and KML feeds. Survey of India announced recently the launch of WMS service of 1:50k scale maps.
Other notable government agencies dealing with mapping are Centers of Department of Space, Survey of India and its units, National Atlas and Thematic Mapping Organization, National Institute of Hydrology etc. Each of the organization host websites to display the content they offer. Primarily, the websites are pure server side web application implementations providing capabilities to the user to search through their catalogues, read metadata and request or order for data. Some of them provide online map viewers that are theme based and selection of data is a wizard driven approach. The hosted map viewers are also primarily server side rendering applications. Initiatives towards making the geo-spatial content availability as web map tiled services (like WMTS and WFS etc.) and associated data in the standard form of JSON / XML etc. and APIs for providing functionality are necessary for clients to use them in building useful mashups.
Also, notable agencies that provide related ancillary data are indiastat.com, censusindia.gov.in etc. that provide vital socio-economic, financial, crime, health and other related data. At present the agencies offer data, hosted through their websites, in HTML table form, Excel (xls format) file download (for paid subscribers). Such data needs to be available in standard GeoRSS, GeoJSON or KML formats for users to quickly pickup required information and building sophisticated map mashup applications. Figure-6 below shows earth quake feed data from IMD overlaid on Bhuvan 3D.
Fig-6 Earthquake feed from IMD overlaid on ISRO Bhuvan
Observations and conclusions
The work reported in this paper was aimed at understanding the capabilities offered and bringing out intricacies rather than the data being used. Emphasis was given to non-programmer’s perspective as it would definitely benefit scientific or domain experts to quickly build and extract useful information. The understanding gained through initial experimentation was that interesting mashup applications can be built using piping or wiring models to intermix or transform the input data into the suitable output forms.
In regard to GIS mashups, referring to experiment made in task1, it was observed that most of online map data providers depict base map in categories such as maps, satellite imagery, terrain, street level etc with very exhaustive place name contextual information. It is interesting to know that most of them provide exhaustive web services API to integrate their maps with client-side applications. Though this is useful it suffices the needs of a casual or a general user who may want to show map data as a background. The requirements for a GIS mashup builder are different in a way that the user may want to add their own data (GIS layers or web map services) upon the base maps that is to be further queried upon, change the symbology as needed, build a query and analysis system and finally host it as a service to share it among the users. In this context, public cloud based GIS prove to be a suitable environment for non-programmers to quickly leverage the advantages of build, develop and deploy.
In regard to other tasks of extracting information from static web sites, aggregating feed data and filter according to user requirements, several client-side tools exist to perform such tasks which otherwise involved good programming experience. Today, certain visual programming based wiring or piping techniques have evolved that makes certain mundane tasks easier to perform and also offers advantages of customizing with modified or new data sources without reprogramming the underlying logic. Such type of client-side mashup developments are ideal for domain experts to extract the important information regarding the subject dealt with rather than worrying about programming techniques.
- Mashup API’s and resources, https://www.programmableweb.com
- Bhuvan, ISRO, India – https://bhuvan.nrsc.gov.in
- GISCloud, https://www.giscloud.com
- ArcGIS, ESRI – https://www.arcgis.com
- Mapbuilder, https://www.mapbuilder.net
- Yahoo Pipes,
- Dapper, https://open.dapper.com
- Ushahidi, https://www.ushahidi.com
- Havaria Information Services – AlertMap, https://hisz.rsoe.hu/alertmap/
- Live Earthquake mashup,
- Open-mashups Studio https://www.open-mashups.org
- Agnes koschimider, Victoria Torres, Vicente Pelechano, Elucidating the mashup Hype: Definition, challenges, methodical guide and tools for mashups: 2nd workshop on mashups, Enterprise mashups and light weight composition on the web in conjunction with the 18th International world wide web conference, pp, 1-9 (2009).
- Wael Al Sarraj and Olga De Troyer. Web Mashup Makers for Casual Users: A User Experiment. International Conference on Information Integration and Web-based Applications & Services (iiWAS 2010) Proceedings, ACM Journal, 2010
- Michael Batty, Andrew Hudson-Smith, Richard Milton & Andrew Crooks (2010): Map mashups, web 2.0 and GIS revolution, Annals of GIS, Vol. 16, No.1,1-13, March 2010.