Skip to content

Latest commit

 

History

History
594 lines (431 loc) · 24.3 KB

DEVELOP.md

File metadata and controls

594 lines (431 loc) · 24.3 KB
Raw

Contribute to the development of ProjetDuMois.fr

Dependencies

Osmium building

ProjetDuMois requires a recent version of osmium as it takes advantage to newest tags-filter abilities. Not many Linux distros got the appropriate package available in their repositories and you may need to build your own binary of osmium.

See guidelines on the official README.

Following packets on debian can be useful

  • build-essential
  • cmake
  • zlib1g-dev
  • libbz2-dev
  • liblz4-dev
  • libboost-dev
  • libboost-program-options-dev

Getting started

git clone https://github.com/vdct/ProjetDuMois.git
cd ProjetDuMois
git submodule update --init

General configuration

The general configuration of the tool is to be filled in config.json. There is a suggested model in the config.example.json file. The parameters are as follows:

  • OSM_USER: OpenStreetMap username for retrieving the modification history with metadata
  • OSM_PASS: password associated with the OSM user account
  • OSM_CLIENT_ID : client ID généré depuis le compte OpenStreetMap
  • OSH_PBF_URL: URL of the OSH.PBF file (history and metadata, example https://osm-internal.download.geofabrik.de/europe/france/reunion-internal.osh.pbf)
  • DB_USE_IMPOSM_UPDATE : enable or disabled Imposm3 integration (to use an existing database which would be maintained by other means, by default true)
  • WORK_DIR: download and temporary storage folder (must have capacity to store the OSH PBF file, example /tmp/pdm)
  • OSM_URL: OpenStreetMap instance to use (example https://www.openstreetmap.org)
  • OSM_API_URL : API OpenStreetMap instance to use (example https://www.api.openstreetmap.org)
  • JOSM_REMOTE_URL: address of the JOSM server to reach (example http://localhost:8111)
  • OSMOSE_URL: Osmose instance to use (example https://osmose.openstreetmap.fr)
  • NOMINATIM_URL: instance of Nominatim to use (example https://nominatim.openstreetmap.org)
  • MAPILLARY_URL: Mapillary instance to use (example https://www.mapillary.com)
  • MAPILLARY_API_KEY: Mapillary API key (could be created here)
  • MATOMO_HOST: URL to the Matomo host you want to report analytics to, with trailing /. Remove to disable. (example https://stats.home.com/).
  • MATOMO_SITE: ID of the Matomo website you want to report analytics (example 6)
  • REPOSITORY_URL: URL of the software repository (example https://github.com/vdct/ProjetDuMois)
  • VECT_STYLE : URL to Maplibre GL compatible style (example https://tile-vect.openstreetmap.fr/styles/liberty/style.json)
  • PDM_TILES_URL: URL to access the pg_tileserv service, which provides the layers in your database
  • GEOJSON_BOUNDS: object of Geometry type (polygon or multipolygon) in GeoJSON delimiting the area to search for OSM notes.
  • MAINTENANCE_MODE: optional flag to display a notice page on website when doing maintenance (set to true to enable)

Postgresql connection

As to connect to any Postgresql host, DB_URL environement variable is expected to be set with a conninfo string. This is necessary for standalone or Docker environments.

export DB_URL="postgres://user:password@host:5432/database"

See also 33.1.1 chapter about Postgresql conninfo strings.

Project configuration

Each project is defined via a subdirectory of `projects'. Each subdirectory must contain the following files :

  • info.json : project metadata
  • howto.md: description of tasks to be performed in Markdown format (use title levels >= 3)
  • contribs.sql : SQL script containing UPDATE request on pdm_changes table, to set contribution classes to certain type of OSM changes and associate points

The properties in info.json are as follows:

  • id: mission identifier (authorized characters: A-Z, 0-9, _ and -)
  • title: name of the mission (short enough)
  • start_date: start date of the mission (format YYYYY-MM-DD)
  • end_date: end date of the mission (format YYYYY-MM-DD)
  • summary: summary of the mission
  • links: definition of the URLs for links to third party pages (OSM wiki)
  • database.osmium_tag_filter : Osmium filter on the tags to be applied to keep only the relevant OSM objects (for example nwr/*:covid19, syntax described here). It is possible to list many filters using & character and same syntax. Only latest defined filter will be used for Osmium feature counts.
  • database.imposm: configuration for importing updated OSM data (types for geometry types to be taken into account, mapping for attributes, see the Imposm documentation for the format of these fields)
  • database.compare: configuration for the search of OpenStreetMap objects to compare, follows the format of database.imposm with an additional property radius (reconciliation radius in meters)
  • datasources: list of data sources that appear on the page (see below)
  • statistics: configuration of statistics display on the project page
  • statistics.count: enable object counting in OSM
  • statistics.feature_name: name to display to the user for these objects
  • statistics.osmose_tasks: name of the tasks performed via Osmose
  • statistics.points: configuration of the points obtained according to the type of contribution (in relation with contribs.sql)
  • editors: specific configuration to each OSM editor. ProjetDuMois is described below, for iD, it is possible to use the parameters listed here.

Projects timing

It is possible to define projects occuring during overlapping time periods. The project:update script will only update currently active projects.

Disable imposm3 usage

It is possible to not use Imposm3 and connect to an existing database already populated with necessary data. You should make sure that it is correctly hourly-updated for this application needs.

In case Imposm3 is disabled, you have also to make available materialized views named pdm_project_${project_id}, with following structure:

osm_id BIGINT
name VARCHAR(255)
tags json
geom GEOMETRY

Optionally, if compare mode is enabled in a given project, another view pdm_project_${project_id}_compare containing data to which features should be compared is necessary. It has the same structure as described above.

In complement of these tables, you need a pdm_boundary table with administrative boundaries for your area (administrative levels 4, 6 and 8) with following structure:

id INT
osm_id BIGINT
name VARCHAR
admin_level INT
tags HSTORE
geom GEOMETRY(Geometry, 3857)
centre GEOMETRY(Point, 3857)

centre column is understood as a point included in the boundary shape (you can use ST_PointOnSurface)

Create indexes on osm_id, tags, geom and centre columns might be useful depending of your database content.

PdM will autonomously derivate a pdm_boundary_subdivide table with usage of ST_Subdivide function as to improve features intersection with admin boundaries.

Data sources

Several data sources can be used, and are to be displayed in the datasources field of the info.json file.

Osmose

Osmose is a tool for quality analysis and open data integration. The properties to be filled in are the following:

  • source (mandatory osmose): source type
  • item: item number (four-digit code)
  • class (optional): class number (multi-digit code)
  • country (optional): Osmose country name pattern (example france*)
  • name: name to be displayed to the user
  • subtitles (optional): key object > value to replace the subtitles of Osmose reports (search by pattern)
  • buttons: label to be displayed on the edit buttons (example { "done": "It's done", "false": "Nothing here" })
  • minzoom (default 7): minimum zoom level for making this layer visible
  • maxzoom (default 18): maximum zoom level for making this layer visible
  • tiles (default): TMS URL list

OSM Notes

The OpenStreetMap notes are a simple method for sending text comments on the map, and facilitate contribution by novice audiences. The properties to be filled in are the following:

  • source (mandatory notes): source type
  • name: name to be displayed to the user
  • description: descriptive text explaining the resolution method for a note
  • terms: list of terms to search for in the notes (singular)
  • buttons: label to display on the edit buttons (example { "close": "It's done" })
  • data (default): data in Geojson format

OpenStreetMap objects

Objects currently present in OpenStreetMap can be displayed to avoid duplicates and allow editing. The properties to be filled in are the following:

  • source (mandatory osm): source type
  • name: name to be displayed to the user
  • description: descriptive text of the displayed object
  • minzoom (default 7): minimum zoom level for making this layer visible
  • maxzoom (default 14): maximum zoom level for making this layer visible
  • tiles (default): TMS URL list
  • layers (default): Layer names list to use and corresponding to tiles indices

This source can appear only once, and corresponds to the objects searched for in the database options of info.json.

OpenStreetMap objects for comparison

Objects indirectly related to the project but relevant to the contribution may also appear. The properties to be filled in are the following:

  • source (mandatory osm-compare): type of source
  • name: name to be displayed to the user
  • description: descriptive text of the displayed object
  • minzoom (default 9): minimum zoom level for making this layer visible
  • maxzoom (default 14): maximum zoom level for making this layer visible
  • tiles (default): TMS URL list
  • layers (default): Layer names list to use and corresponding to tiles indices

This source can only appear once, and corresponds to the objects searched for in the database.compare options of info.json.

OpenStreetMap extra objects

Objects outside the project's scope, displayed as to inform contributors that something different already exists in place. The properties to be filled in are the following:

  • source (mandatory osm-extra): type of source
  • name: name to be displayed to the user
  • description: descriptive text of the displayed object
  • minzoom (default 9): minimum zoom level for making this layer visible
  • maxzoom (default 14): maximum zoom level for making this layer visible
  • tiles (default): TMS URL list
  • layers (default): Layer names list to use and corresponding to tiles indices

This source can appear as many time as required

Background imagery

Raster tile imagery can be added in background to make contribution easier or give context. You have to define following properties:

  • source (mandatory background): type of source
  • icon (default other): symbol to display in legend (between aerial, thematic, picture, other)
  • name: name shown to users
  • tiles (default): list of TMS URL
  • attribution: attribution to display on map
  • minzoom (default 2): minimum zoom level for making this layer visible
  • maxzoom (default 19): maximum zoom level for making this layer visible
  • tileSize (default 256): width and length of a tile in pixels

These sources should be declared in reverse order of display. The lower layer should be declared first.

Sources stats

Another kind of datasource can be added and refers to geographical statistics, over administrative boundaries for instance

  • source (mandatory stats): statistics source type
  • minzoom (default 2): minimum zoom level for making this layer visible
  • maxzoom (default 14): maximum zoom level for making this layer visible
  • tiles (default): list of TMS URL
  • layers (default): Layer names list to use and corresponding to tiles indices

Projet du Mois integrated editor

Project configuration allows to customize embedded editor with appropriate fields. Let's consider following json to be set in the editors list:

"pdm": {
  "fields": [
    ... fields list...
  ],
  "title": {
    "add": "Add action label",
    "edit": "Edit action label"
  }
}

Fields are defined with standard JSON objects added to the fields array in the uspide json. Every type except hidden supports those common attributes:

  • name: Field name to be displayed to the user
  • help: HREF link to a relevant help page regarding this particular field
  • description: An extensive text displayed to the user providing details about the field
  • optional: A boolean false/true field making the field respectively mandatory or not

Static or external attributes

Define static tags to be added to every created object by the editor, or retrieve existing values from external sources (like identifiers from Osmose such as ref:FR:SIRET tag, use in that case * value).

{ "type": "hidden", "tags": { "tag_1": "value_1", "tag_2": "value_2", "external_tag_3": "*" } }

Scalar field

Text fields comes with 3 different types: text, number or email. They're all producing a standard text field, leading to a given OSM key with appropriate validation features.

  { "type": "text", "name": "Field label", "tag": "tag_key", "optional": false },
  { "type": "number", "name": "Field label", "tag": "tag_key", "optional": false },
  { "type": "email", "name": "Field label", "tag": "tag_key", "optional": false }

Textarea field

A wider textarea field similar to text one.

{ "type": "textarea", "name": "Field label", "tag": "tag_key", "optional": false }

Drop down list

A drop down list with custom entries leading to a given OSM tag

{
  "type": "select",
  "name": "Field label",
  "tag": "tag_key",
  "optional": false,
  "values": [
    { "v": "value_1", "l": "Value 1 label" },
    { "v": "value_2", "l": "Value 2 label" }
  ]
}

You can also set multiple tags using a single value in the list, for example:

{
  "type": "select",
  "name": "Type",
  "tag": "_select1",
  "values": [
    {
      "l": "National police",
      "tags": { "name": "National police", "operator": "National police", "police:FR": "police" }
    },
    { "l": "City police", "tags": { "name": "City police", "police:FR": "police_municipale" } }
  ]
}

In case of multiple tags for a single value, you can force removal of certain tags with empty strings, for example to cover objects that belongs to different categories:

{
  "type": "select",
  "name": "Type de commerce",
  "tag": "_select1",
  "values": [
    { "l": "Agence d'assurance", "tags": { "shop": "", "office": "insurance" } },
    { "l": "Agence de voyages", "tags": { "shop": "travel_agency", "office": "" } }
  ]
}

2 or 3 states

States input are using radio buttons to provide 2 or 3 options to the user. It leads to a given OSM key. 2states is yes/unknown and 3states is yes/no/unknown.

  { "type": "2states", "name": "Field label", "tag": "tag_key"},
  { "type": "3states", "name": "Field label", "tag": "tag_key"}

Brands and operators

To make brand or operators input easier, you can use the nsi field (related to Name Suggestion Index, a collaborative listing of all brands and operators worldwide). It has specific options:

  • path : path to brand listing to use (this appears in website title, for example brands/shop/coffee)
  • locationSet : a two-letter, lowercase country code to only list brands from this specific country (optional)

For example:

{ "type": "nsi", "name": "Brand", "path": "brands/shop/bakery", "locationSet": "fr" }

Icons group

Icon select fields allow simple selecting of several similar attributes, for example available sockets at a charging station or kind of waste accepted in a recycling container. Icons must be available in the /website/images/form folder, as PNG image. Tags associated to an icon are applied on feature if icon is selected.

{
  "type": "icons",
  "name": "Available sockets",
  "tag": "_socket",
  "values": [
    { "icon": "socket_typee", "label": "TE", "tags": { "socket:typee": "yes" } },
    { "icon": "socket_type2", "label": "T2", "tags": { "socket:type2": "yes" } },
    { "icon": "socket_type2_combo", "label": "T2 Combo", "tags": { "socket:type2_combo": "yes" } },
    { "icon": "socket_chademo", "label": "Chademo", "tags": { "socket:chademo": "yes" } }
  ]
}

Feature counts and statistics

Project statistics are made by ./db/31_projects_update_tmp.sh script. This script fills pdm_feature_counts SQL table with missing daily data according to last OSH file timestamp and current day.

It is possible to force full recount for a project by deleting OSH timestamp file, retreive again PBF/PBH files and launch again the script:

rm ${WORK_DIR}/osh_timestamp
./db/11_pbf_update_tmp.sh
./db/31_projects_update_tmp.sh

Points and contributions

Certain OSM contributions can give points to users. Each project configuration set how many points are given according to the type of contribution. By default, the platform create the following contribution types:

  • add: changes concerning features with version=1 (creation)
  • edit : changes concerning features version>1 (tag or geometry edits)

It is possible to attribute your own type for each project by creating a contribs.sql file next to info.json. This script contains UPDATE SQL requests to add entries in pdm_changes table. Each OSM change can only have a single type and have a single amount of points associated.

Configuration of points is in info.json:

{
  "statistics": {
    "points": { "add": 3, "project1": 1 }
  }
}

Build

Once PDM has been properly configured, you should choose between Docker or standalone to build it. Refers to Database section in run chapter to make ProjetDuMois fully runable.

git submodules

ProjetDuMois relies on some git submodules. Please mind using the following to retreive them prior to build

git submodule init
git submodule update

Docker build

You can build a node.js based ProjetDuMois server including necessary features to run. It includes osmium 1.10.0 with Debian Buster. It doesn't includes a PgSQL server. You can use CampToCamp Postgres image.

docker build [--build-arg IMPOSM3_VERSION=0.11.0] -t pdm/server:latest .

Where:

  • IMPOSM3_VERSION: Version of imposm3 to use in the docker image

Standalone instance

npm install

Run

Database

The database relies on PostgreSQL. To create the database :

psql -c "CREATE DATABASE pdm"

pg_tileserv

Without Docker

As pg_tileserv is required to display osm data on the map, you'll have to install it or use the image include in docker-compose file. You will need wget and unzip to use it.

# download latest build
wget https://postgisftw.s3.amazonaws.com/pg_tileserv_latest_linux.zip
# unzip / delete it
unzip pg_tileserv_latest_linux.zip -d /opt/pg_tileserv
rm pg_tileserv_latest_linux.zip
# specify database url to connect
export DATABASE_URL=postgres://postgres:password@pdm-db:5432/pdm
# launch it
cd /opt/pg_tileserv
./pg_tileserv

Using Docker

You can use the image in docker-compose.yml or for a lightweight image you should build yourself the alpine Dockerfile It's possible to install pg_tileserv in pdm Dockerfile before building the image. But it's not recommended.

Docker

Database is installed and inited simply with:

docker run --rm [--network=your-network] -e DB_URL=postgres://user:password@host:5432/database pdm/server:latest install
docker run --rm [--network=your-network] -v host_work_dir:container_work_dir -e DB_URL=postgres://user:password@host:5432/database pdm/server:latest init

And then run the server with:

docker run -d --rm [--network=your-network] -p 3000:3000 --name=pdm -v host_work_dir:container_work_dir -e DB_URL=postgres://user:password@host:5432/database pdm/server:latest run

Don't forger to add following lines into your crontab for periodic updates, for example daily updates:

docker run --rm [--network=your-network] -v host_work_dir:container_work_dir -e DB_URL=postgres://user:password@host:5432/database pdm/server:latest update_daily

Individual updates are also available for punctual calls:

docker run --rm [--network=your-network] -v host_work_dir:container_work_dir -e DB_URL=postgres://user:password@host:5432/database pdm/server:latest update_pbf
docker run --rm [--network=your-network] -v host_work_dir:container_work_dir -e DB_URL=postgres://user:password@host:5432/database pdm/server:latest update_features
docker run --rm [--network=your-network] -v host_work_dir:container_work_dir -e DB_URL=postgres://user:password@host:5432/database pdm/server:latest update_projects

Standalone

Database relies on PostgreSQL. To install the schema before first run:

psql -d pdm -f db/00_init.sql

The following script is to run to retreive and update PBF/PBH files:

npm run pbf:update
./db/11_pbf_update_tmp.sh

The following script is to run after first initialization of database to create list of OSM features:

npm run features:update
./db/21_features_update_tmp.sh init

The following script has to be launched daily to retrieve the contribution statistics (notes, objects added, badges obtained):

npm run projects:update
./db/31_projects_update_tmp.sh

Website

The code for the web interface can be found in the website folder. This is an ExpressJS server, combined with Pug templates.

The Pug templates are in the templates sub-folder. It is organized according to the following logic:

  • In templates, the general model layout.pug and its CSS file
  • In common, generic elements to all pages (<head>, header, footer)
  • In components, the main components that populate the pages (map, statistics block...)
  • In pages, each page of the site (home, map, project page...)

The site can be viewed at localhost:3000.

Docker

Docker image includes websites and background updating tasks. You can run it with:

docker run -d --rm [--network=your-network] -p 3000:3000 --name=pdm -v host_work_dir:container_work_dir -e DB_URL=postgres://user:password@host:5432/database pdm/server:latest run

Docker compose

A compose file is provided to ease the running processing. It won't prevent you from creating database, adding users and make the appropriate configuration nor building dockers as mentionned upside. Docker compose only allows to run easilly a functionnal instance if and only it has already been properly configured before.

Depending on your Postgresql configuration, you'll surely have to customize the DB_URL env variable in the compose file to let the pdm server access the database safely.

Don't try to run the instance with docker-compose first, try to run each component separately and check if everything work as expected. Once everything runs normally, you can use the following for further runs:

To start:

docker-compose up

To stop:

docker-compose down

Standalone

Standalone running requires a node server complient with compatility at the top of this document and planned tasks to update projects regularly.

To launch the web site :

export DB_URL="postgres://user:password@host:5432/database" # Database URL
export PORT=3000 # Nodejs port (defaults to 3000)
npm run start

The site can be viewed at localhost:3000.