OSM Export Tool

Osm Export Tool platform allows you to create custom OpenStreetMap exports for various HOT regions. You can specify an area of interest and a list of features (OpenStreetMap tags) for the export. A current OpenStreetMap data extract for that area in various data formats is then created for you within minutes.

The live site http://export.hotosm.org is currently still powered by the older version 1 code living at https://github.com/hotosm/hot-exports

This repo contains the newly re-written version 2 of the OSM exports tool.

Installation Instructions

Some prior experience with Django would be helpful, but not strictly necessary.

Update Packages

$ sudo apt-get update
$ sudo apt-get upgrade

Python

HOT Exports requires Python 2.7.x.

pip

To install pip, run:

$ sudo apt-get install python-pip

Git

Git is used for version control. To check your version, and if you have it installed:

$ git --version or run $ sudo apt-get install git

Virtualenv

Virtualenv (virtual environment) creates self-contained environments to prevent different versions of python libraries/packages from conflicting with each other.

To make your life easier install virtualenvwrapper

$sudo pip install virtualenvwrapper

Add the following to .bashrc or .profile

export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/dev
source /usr/local/bin/virtualenvwrapper.sh

Run source ~/.bashrc

Run mkvirtualenv hotosm to create the hotosm virtual environment.

Change to the $HOME/dev/hotosm directory and run workon hotosm.

Postgres

Install PostgreSQL / PostGIS and its dependencies,

$ sudo apt-get install libpq-dev python-dev

$ sudo apt-get install postgresql postgresql-contrib

$ sudo apt-get install postgis postgresql-9.3-postgis-2.1

Create the database and role

$ sudo su - postgres
$ createdb 'hot_exports_dev'
$ create role hot with password '<-password->'

You might need to update the pg_hba.conf file to allow localhost connections via tcp/ip or allow trusted connections from localhost.

Run $ psql -h localhost -U hot -W hot_exports_dev

$ ALTER ROLE hot SUPERUSER;
$ ALTER ROLE hot WITH LOGIN;
$ GRANT ALL PRIVILEGES ON DATABASE hot_exports_dev TO hot;
$ CREATE EXTENSION POSTGIS;
$ CREATE EXTENSION HSTORE;

Create the exports schema

$ CREATE SCHEMA exports AUTHORIZATION hot;

Install GDAL

$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt-get update
$ sudo apt-get install gdal-bin libgdal-dev

To install the python GDAL bindings into your virtualenv you need to tell pip where to find the libgdal header files, so in your shell run:

$ export CPLUS_INCLUDE_PATH=/usr/include/gdal
$ export C_INCLUDE_PATH=/usr/include/gdal

Install third-party dependencies

The HOT Export pipeline depends on a number of third-party tools.

$ sudo apt-get install osmctools

$ sudo apt-get install spatialite-bin libspatialite5 libspatialite-dev

$ sudo apt-get install default-jre zip unzip

Garmin

Download the latest version of the mkgmap utility for making garmin IMG files from http://www.mkgmap.org.uk/download/mkgmap.html

Download the latest version of the splitter utility for splitting larger osm files into tiles. http://www.mkgmap.org.uk/download/splitter.html

Create a directory and unpack the mkgmap and splitter archives into it.

OSMAnd OBF

For details on the OSMAnd Map Creator utility see http://wiki.openstreetmap.org/wiki/OsmAndMapCreator

Download the OSMAnd MapCreator from http://download.osmand.net/latest-night-build/OsmAndMapCreator-main.zip. Unpack this into a directory somewhere.

Install RabbitMQ

HOT Exports depends on the rabbitmq-server. For more detailed installation instructions see http://www.rabbitmq.com/install-debian.html. The default configuration should be fine for development purposes.

$ sudo apt-get install rabbitmq-server

Checkout the HOT Export Tool source

In the hotosm project directory run:

$ git clone git@github.com:hotosm/osm-export-tool2.git

Install the project's python dependencies

From the project directory, install the dependencies into your virtualenv:

$ pip install -r requirements-dev.txt

or

$ pip install -r requirements.txt

Project Settings

Create a copy of hot_exports/settings/dev_dodobas.py and update to reflect your development environment.

Make sure the following configuration variables are set:

Set OSMAND_MAP_CREATOR_DIR to the directory where you installed the OSMAnd MapCreator.

Set GARMIN_CONFIG to point to the absolute path to the garmin configuration file, ./utils/conf/garmin_config.xml by default.

Update the utils/conf/garmin_config.xml file. Update the garmin and splitter elements to point to the absolute location of the mkgmap.jar and splitter.jar utilites.

Once you've got all the dependencies installed, run ./manage.py migrate to set up the database tables etc.. Then run ./manage.py runserver to run the server. You should then be able to browse to http://localhost:8000/

Overpass API

The HOT Exports service uses a local instance of Overpass v07.52 for data extraction. Detailed instructions for installing Overpass are available here.

Download a (latest) planet pbf file from (for example) http://ftp.heanet.ie/mirrors/openstreetmap.org/pbf/.

To prime the database we've used osmconvert as follows:

osmconvert --out-osm planet-latest.osm.pbf | ./update_database --meta --db-dir=$DBDIR --flush-size=1

If the dispatcher fails to start, check for, and remove osm3s_v0.7.52_osm_base from /dev/shm.

We apply minutely updates as per Overpass installation instructions.

Celery Workers

HOT Exports depends on the Celery distributed task queue. As export jobs are created they are pushed to a Celery Worker for processing. At least two celery workers need to be started as follows:

From a 'hotosm' virtualenv directory (use screen), run:

export DJANGO_SETTINGS_MODULE=hot_exports.settings.your_settings_module

$ celery -A hot_exports worker --loglevel debug --logfile=celery.log.

This will start a celery worker which will process export tasks. An additional celery worker needs to be started to handle purging of expired unpublished export jobs. From another hotosm virtualenv terminal session, run:

export DJANGO_SETTINGS_MODULE=hot_exports.settings.your_settings_module

$ celery -A hot_exports beat --loglevel debug --logfile=celery-beat.log

See the CELERYBEAT_SCHEDULE setting in settings.py.

For more detailed information on Celery Workers see here

For help with daemonizing Celery workers see here

Front end tools

HOT Exports uses bower to manage javascript dependencies.

sudo apt-get install nodejs
sudo apt-get install npm
sudo npm -g install bower
sudo npm -g install yuglify

Using Transifex service

To work with Transifex you need to create ~/.transifexrc, and modify it's access privileges

chmod 600 ~/.transifexrc

Example .transifexrc file:

[https://www.transifex.com]
hostname = https://www.transifex.com
password = my_super_password
token =
username = my_transifex_username

Managing source files

To update source language (English) for Django templates run:

python manage.py makemessages -l en

To update source language for javascript files run:

python manage.py makemessages -d djangojs -l en

then, push the new source files to the Transifex service, it will overwrite the current source files

tx push -s

Pulling latest changes from the Transfex

When adding a new language, it's resource file does not exist in the project, but it's ok as it will be automatically created when pulling new translations from the service. To add a local mapping:

tx set -r osm-export-tool2.master -l hr locales/hr/LC_MESSAGES/django.po

or for javascript files:

tx set -r osm-export-tool2.djangojs -l hr locales/hr/LC_MESSAGES/djangojs.po

Once there are some translation updates, pull the latest changes for mapped resources

For a specific language(s):

tx pull -l fr,hr

For all languages:

tx pull

Finally, compile language files

python manage.py compilemessages