happygisco

Simple geoservice interface (API) on top of Eurostat GISCO web-services.

The project happyGISCO (pronounce as if you were French) provides with the implementation of a Python interface to GISCO web-services. The module happygisco will enable you to:

• run some of the basic geographical operations supported by GISCO, e.g. geocoding, routing and NUTS identification,
• retrieve most of the datasets (e.g., vector layers of countries, NUTS, ...) made available through GISCO Rest API.

documentation	available at: http://happygisco.readthedocs.io
status	since 2018 – in construction
contributors
license	EUPL

This material accompanies the articles referenced below and illustrates the idea of Eurostat data as a service. The rationale is further described in the paper "Empowering and interacting with statistical produsers: A practical example with Eurostat data as a service".

Quick install and start

TBC

Once installed, the module can be imported simply:

>>> import happygisco

Notebook examples

Simple examples are available in the form of Jupyter notebooks under the notebooks/ folder, e.g.:

• some basic calls to the geocoding services,
• a simple application with NUTS vector features,
• an efficient tiled-based and multiprocessor algorithm for the overlay of vector model layer onto a regular grid, with NUTS polygonal representation as an illustration.
• an extended workflow for location identification and retrieval.

Usage

Services

Some variants of the geolocation service are made available through the implementation of different classes:

• OSMService: this is an interface to OpenStreetMap native geocoding and routing web-services;
• GISCOService: this is an interface to Eurostat GISCO web-services; the geocoding and routing tools are also based on OpenStreetMap (the class GISCOService derives from OSMService); it also enables the users to retrieve the NUTS region at any level from any geolocation given by its toponame (place) or its geographical coordinates;
• APIService: this calls other "external" geo- web-services (including Google maps), e.g. to geolocate geographical features.

Note that no caching is performed when running the services, unless they are run from one of the features instance below (e.g. Location).

It is pretty straigthforward to create an instance of a service, for example GISCOService to call GISCO web-services:

>>> from happygisco import services
>>> service = services.GISCOService()

and run the supported methods:

>>> place = "Lampedusa, Italia"
>>> coord = service.place2coord(place, unique=True)
>>> print(coord)
    [35.511134150000004, 12.59629135962961]
>>> alt_place = service.coord2place(coord)
>>> print(alt_place)
    'Strada di Ponente, Lampedusa e Linosa, (Sicily), Italy'
>>> nuts = service.coord2nuts(coord, level=2)
>>> print(nuts)
    {'attributes': {'CNTR_CODE': 'IT', 'LEVL_CODE': '2', 'NAME_LATN': 'Sicilia',
     'NUTS_ID': 'ITG1',  'NUTS_NAME': 'Sicilia',  'OBJECTID': '320',
     'SHRT_ENGL': 'Italy'},
     'displayFieldName': 'NUTS_ID',
     'layerId': 2, 'layerName': 'NUTS_2013', 'value': 'ITG1'}

Note that, in order to make things easier, it is possible to parse lists of places instead of single places:

>>> axis = ['Rome, Italy', 'Berlin, Germany', 'Tokyo, Japan']
>>> for p in axis:  # either iterating over the places
...     print(service.place2coord(p, unique=True))
    [41.8933203, 12.4829321]
    [52.5170365, 13.3888599]
    [34.6968642, 139.4049033]
>>> coord = service.place2coord(axis, unique=True) # or running the method for the whole list
>>> print(coord)
    [[41.8933203, 12.4829321], [52.5170365, 13.3888599], [34.6968642, 139.4049033]]
>>> service.coord2nuts(coord, level=2)
    [{'attributes': {'CNTR_CODE': 'IT', 'LEVL_CODE': '2', 'NAME_LATN': 'Lazio',
      'NUTS_ID': 'ITI4', 'NUTS_NAME': 'Lazio', 'OBJECTID': '330',
      'SHRT_ENGL': 'Italy'},
      'displayFieldName': 'NUTS_ID',
      'layerId': 2, 'layerName': 'NUTS_2013', 'value': 'ITI4'},
     {'attributes': {'CNTR_CODE': 'DE', 'LEVL_CODE': '2', 'NAME_LATN': 'Berlin',
      'NUTS_ID': 'DE30', 'NUTS_NAME': 'Berlin', 'OBJECTID': '202',
      'SHRT_ENGL': 'Germany'},
      'displayFieldName': 'NUTS_ID',
      'layerId': 2, 'layerName': 'NUTS_2013', 'value': 'DE30'},
     None]

You are also offered to use other geo web-services using APIService, e.g. any of those listed below:

>>> print(APIService.AVAILABLE)
   ['GMaps', 'OpenMapQuest', 'YahooPlaceFinder', 'LiveAddress', 'Bing', 'GeoNames', 'GoogleV3', 'Nominatim', 'MapQuest'] 

Depending on the service selected, you may be requested to provide with your own credentials:

>>> service = services.APIService(coder='Nominatim') # no key required
>>> service.place2coord('Paris, France')
    [48.8566101, 2.3514992]
>>> service = services.APIService(coder='GMaps', key='???') # use your own key here
>>> service.place2coord('Paris, France')
    [48.856614, 2.3522219]
>>> service = services.APIService(coder='GeoNames', username='???') # use your own username here
>>> service.place2coord('Paris, France')
    [48.85341, 2.3488]

Features

It is possible to create simple geographical features whose methods implement and apply the different services defined above, e.g.:

• a Location: a feature representing a geolocation, i.e. defined as a topo/placename or as a list of geographical coordinates,
• an Area: a simple vector geometry () in the sense of GISCO services expressed as a dictionary, i.e., structured like the JSON file returned by the GISCO geocoding or reverse geocoding services,
• a NUTS: the vector geometry representing a NUTS area expressed as a dictionary, i.e., structured like the JSON file returned by the GISCO findnuts services.

One can for instance declare a specific location, and run any of the methods supported by the Location class:

>>> from happygisco import features
>>> location = features.Location(place="Lisbon, Portugal")
>>> location.coord
    [38.7077507, -9.1365919]
>>> location.routing('Paris, France')
    ({'distance': 3058767.9, 'duration': 377538.2,
      'geometry': 'uv}qEaeqhEo_XlbOutDa`~@uuVocZqa|@ttDqaZneRwcjEetxBwfYags@}_nAugsAmaYcmcApxCiiuDcvi@webB`dFeix@q}VqdvAfaj@greAtqEuwi@c~QmvqCuhZ}o`AzzVkv{@egOo|Vjf@avyCrlZocsFwo_@ef`DgdKkqQ{gPbkA{pUgwq@h{[s}`B`hJsgnBaq^oMetAkab@q~j@at~@hbd@yheAhmh@gad@vyz@dit@uxz@kjt@knh@lbd@ibd@xheAp~j@`t~@dtAjab@`q^nMahJrgnBe|[x}`BvqU`wq@nkPsgAt_KlnQdo_@r}_DwkZlksFkg@joyCdhOjzVk{V|f|@vhZph`Ab~Q`vqCsnEjpi@wdj@tyeAx|Vd`vA_cF~mx@~ui@tebB_yCtguD~aYjocAn`nAhgsAtfYrgs@pdjEbrxBhaZieR~a|@{tD`vV|cZ~}F`_}@nuUaaN',
      'legs': [{'distance': 1530444.4, 'duration': 188741.1, 'steps': [], 'summary': ''},
               {'distance': 1528323.5, 'duration': 188797.1, 'steps': [], 'summary': ''}]},
     [{'hint': 'DcOGgEuuRIQAAAAAAAAAAE0AAAAAAAAASgQAAOofZwBScQAAzuv3AcpmDwImok4CMZZ0_wAAAQEZfn5e',
       'location': [33.024974, 34.563786], 'name': ''},
      {'hint': 'mRIbgp0SG4IAAAAAAAAAAFoAAAAAAAAAogIAADJYZwFScQAAeuyjAgCdNAIifukCi-EjAAAAAQEZfn5e',
       'location': [44.297338, 37.002496], 'name': ''},
      {'hint': 'DcOGgEuuRIQAAAAAAAAAAE0AAAAAAAAASgQAAOofZwBScQAAzuv3AcpmDwLU3csCvrFGAAAAAQEZfn5e',
       'location': [33.024974, 34.563786], 'name': ''}])
>>> location.findnuts(level=[2,3])
     {2: 'PT17', 3: 'PT170'}
>>> location.distance('Paris, France')
    1455.7107037157618

What about creating a NUTS object:

>>> nuts = features.NUTS()

Tools

Geospatial tools are derived from gdal methods and provided in the GDALTransform class.

These tools can be used, for instance, with NUTS appropriate vector data sources to operate the NUTS identification. Note that it is a brute-force solution, since the program will explore sequentially all NUTS features so as to identify the correct region. This could be improved using a multithread process for instance, e.g. using multiprocessing module. Besides, the program does not check the validity of the result returned by Google maps services, since this result can be ambiguous and/or inaccurate.

In the associated classes GeoAngle and GeoCoordinate, you will find also some basic implementations of simple geoprocessing tools, e.g. units conversion, (geodesic) distance calculation, ... For a quick review on the latter, have for instance a look at this.

Data resources

• The Geographic Information System of the Commission at Eurostat: GISCO .
• GISCO webservices: find-nuts and geocode.
• GISCO data distribution REST API and its visualisation tool.
• GISCO themes with links to countries (corresponding list of datasets) and NUTS (corresponding list of datasets).
• NUTS bulk download page. You can for instance download 2013 (1:60 Million) NUTS data and find out how to interpret it.
• NUTS online and NUTS on EU Vocabularies.
• GISCO administrative and statistical units and correspondence table between degree of urbanisation (DEGURBA) or local administrative units.
• TERCET tool and territorial typologies.
• Service Nuts2json of NUTS topojson/geojson datasets reformatted for web-mapping (datasets and scripts).

Software resources/dependencies

• gdal package and cookbook.
• Geocoding/processing packages: googlemaps, googleplaces or geopy (suggested).
• Packages for (geospatial) data handling: pandas and geopandas.
• Packages for map visualisations: ipyleaflet or folium.
• asyncio library and aiohttp for asynchronous I/O.
• Packages for caching: requests_cache or cachecontrol.

References

• Grazzini J., Museux J.-M. and Hahn M. (2018): Empowering and interacting with statistical produsers: A practical example with Eurostat data as a service, in Proc. Conference of European Statistics Stakeholders, doi:10.5281/zenodo.3240557.
• Grazzini J., Lamarche P., Gaffuri J. and Museux J.-M. (2018): "Show me your code, and then I will trust your figures": Towards software-agnostic open algorithms in statistical production, in Proc. Quality Conference, doi:10.5281/zenodo.3240282.
• Downey. A (2012): Think Python, Green Tea Press.