Publishing geospatial data online through REST API.
- Two publication models available:
- layer: visual representation of single vector or raster dataset, including raster timeseries
- map: collection of layers
- Accepts vector layer data in GeoJSON, ShapeFile, or PostGIS table identified by PostgreSQL connection URI
- Accepts raster layer data in GeoTIFF, JPEG2000, PNG, and JPEG formats
- Accepts layer style in Styled Layer Descriptor, Symbology Encoding, and QGIS Style File Format (for vector data only) formats
- Accepts map definition in HSLayers Map Composition format
- Even large files can be easily uploaded from browser thanks to asynchronous chunk upload
- OAuth2 authentication
- Authorization enables to set read and write access to each layer and map for specific users
- Asynchronous processing
- Provides URL endpoints
- Documented security system, data storage, and proxying
- Configurable by environment variables
- Standing on the shoulders of Docker, Python, Flask, PostgreSQL, PostGIS, GDAL, QGIS Server, GeoServer, OpenLayers, Celery, Redis, and more
- Inspired by CCSS-CZ/layman
- at least 3 GB RAM
- at least 5 GB disk space
- Linux kernel v4.11+ (because of QGIS server and its Qt 5.10+/libQt5Core dependency), for example
-
Ubuntu 18.04.2+
-
CentOS 8+
-
Fedora 29+
You can check your kernel version with
uname -r
.
-
- Docker Engine v20.10.13+ including Docker Compose v2+
- installation instructions for centos 7, including docker-compose-plugin installation
Optionally
- linux or any tool to run tasks defined in Makefile using
make
command- can be replaced by running commands defined in Makefile directly
- git
- can be replaced by downloading ZIP archive of the repository
git clone https://github.com/LayerManager/layman.git
cd layman
# checkout the latest release in current branch
git checkout $(git describe --abbrev=0 --tags)
There are following ways how to run Layman:
- demo: the easiest and recommended way how to start layman for demonstration purposes
- development: run layman in development mode
- test: run automatic tests
- production: run layman in production, requires installation of external dependencies and manual configuration
This is the easiest way how to start layman for demonstration purposes. It includes also external dependencies. However it's not meant for production and it's not safe for production. Performance might be also an issue.
# use demo settings
cp .env.demo .env
# start dockerized containers in background
make start-demo-full
Initial startup may take few minutes (download docker images, build it, run it). You are interested in container named layman
. You can check its logs with command
docker logs -f layman
Wait until you see something like
[2019-10-30 13:45:36 +0000] [12] [INFO] Layman successfully started!
Then visit http://localhost/. You will see simple web client that interacts with REST API. To check if Layman is running, call GET Version.
To stop running service, run make stop-demo
.
By default, demo starts without authentication provider. You can publish layers and maps as anonymous user, but you can't log in nor publish any private data.
To be able to use authentication and publish private data, follow these steps:
- Stop demo instance by
make stop-demo
- Copy all
OAUTH_*
environment variables from.env.dev
to.env
file. - In
.env
change environment variables:LAYMAN_AUTHN_MODULES=layman.authn.oauth2,layman.authn.http_header OAUTH2_CALLBACK_URL=http://localhost/client/authn/oauth2-provider/callback
- Start demo with
make start-demo-full-with-optional-deps
Then you can log in with automatically provided Wagtail user layman
and password laymanpwd
.
Layman's source code provides settings suitable for development, testing and demo purposes. Furthermore, there exists Makefile
with predefined commands for each purpose including starting all necessary services (both in background and foreground) and stoping it.
Layman's configuration is split into three levels:
docker-compose.*.yml
files used as docker-compose configuration files with most general settings of docker containers including volume mappings, port mappings, container names and startup commands.env*
files with environment settings of both build stage and runtime of docker containerssrc/layman_settings.py
Python module with settings of Layman's Python modules for runtime
Files at all three levels are suffixed with strings that indicates what they are intended to:
demo
to demonstration purposesdev
to developmenttest
to automatic testsdeps
to external dependencies
When you are switching between different contexts (e.g. between demo and dev), always check that you are using settings intended for your context, especially
.env*
file (checkenv_file
properties indocker-compose.*.yml
file)
Also, anytime you change .env
file, remember to rebuild docker images as some environment variables affect build stage of docker images. Rebuild happens automatically as a part of make start-demo*
commands.
Layman has many dependencies. Most of them are shipped with Layman. However, there are some external dependencies that should be treated carefully:
- PostgreSQL & PostGIS
- QGIS Server
- GeoServer
- Redis
- Micka
These external dependencies are shipped with Layman for development, testing and demo purposes. They are grouped in docker-compose.deps*.yml
files.
However, if you want to run Layman in production, it is strongly recommended to install external dependencies separately. Recommended (i.e. tested) versions are:
- PostgreSQL 13.3 & PostGIS 3.1
- QGIS Server 3.16.1
- GeoServer 2.21.2
- Redis 4.0
- Micka 2020.014 (versions >=2020.010 probably work too)
Layman follows semantic versioning, so any change in MINOR and PATCH version should be backwards compatible. Still, human make mistakes, so it's recommended to backup data directories as a part of each upgrade.
General steps to upgrade layman to MINOR or PATCH version:
- Stop layman. For demo configuration,
make stop-demo
, for dev configurationmake stop-dev
. - Backup data directories. By default, they are located at
layman_data
deps/*/data
(data directories of external dependencies)
- Follow Upgrade requirements in Changelog of all MINOR and PATCH versions greater than your current version and lower or equal to the version you are upgrading to.
- If you run Layman in development mode (e.g. by
make start-dev
), run alsomake build-dev
make client-build
make timgen-build
make wagtail-build
make micka-build
- If you run Layman in development mode (e.g. by
- If you are expecting long-running upgrade, run standalone upgrade, otherwise Gunicorn could time out. The command depends on how you are starting Layman.
- If you are starting Layman with
make start-demo
, runmake upgrade-demo
. - If you are starting Layman with
make start-demo-full
ormake start-demo-full-with-optional-deps
, runmake upgrade-demo-full
. - If you are starting Layman with
make start-dev
, you don't need to run standalone migration.
- If you are starting Layman with
- Start Layman.
NOTE: If upgrade failed due to timeout of request to GeoServer, run
make upgrade-after-timeout
to finish upgrade. NOTE: If you run in problems during upgrade process, especially before v1.12.0, you can try to upgrade gradually through individual minor versions.
To run layman in production, you need to provide external dependencies and configure layman manually.
When providing external dependencies, check their production-related documentation:
- PostgreSQL 13.3 & PostGIS 3.1
- QGIS Server 3.16.1
- GeoServer 2.21.2
- Redis 4.0
- Micka v2020.014, see also configuration of dockerized Micka.
Within PostgreSQL, you need to provide one database for Layman and one database for Micka. For Layman, you also need to provide one user LAYMAN_PG_USER who needs enough privileges to create new schemas in LAYMAN_PG_DBNAME database. The user also needs access to public
schema where PostGIS must be installed.
Within QGIS Server, you do not need to provide anything special.
Within GeoServer, you need to provide either admin password GEOSERVER_ADMIN_PASSWORD, or one Layman user LAYMAN_GS_USER and one layman role LAYMAN_GS_ROLE. If admin password is provided, Layman will create the Layman user and the Layman role automatically. URL path of the GeoServer must be /geoserver/
.
Within Redis, you need to provide two databases, one for Layman, second for Layman Test Client. Connection strings are defined by LAYMAN_REDIS_URL and LTC_REDIS_URL.
Within Micka, you need to provide one user with editor privileges, whose credentials are defined by CSW_BASIC_AUTHN.
After providing external dependencies there is time to provide internal dependencies (system-level, python-level and node.js-level dependencies). You can either use our docker and docker compose configuration to generate docker images that already provides internal dependencies, or you can provide internal dependencies by yourself (if you prefer not to use docker in production).
System-level dependencies includes
- python 3.8+
- python3-lxml
- ogr2ogr utility of gdal 3.3+
- chromium-browser 90+ and corresponding version of chromedriver
- pipenv
- node.js 18 & npm 8 for running Layman Test Client
- node.js 16 & npm 8 for running Timgen
Pipenv is recommended tool for installing python-level dependencies. Both Pipfile and Pipfile.lock are located in docker/
directory.
Npm is recommended tool for installing node.js-level dependencies. Both package.json and package-lock.json are located in timgen/
directory.
Next you need to choose how you deploy Layman. As Layman is Flask application, check Flask's deployment options. Layman is safe to run with multiple WSGI Flask processes and with multiple Celery worker processes.
Configure Layman using environment settings. Demo configuration is a good starting point to setup Layman for production, however it needs to be adjusted carefully.
Last, start layman and necessary services:
- thumbnail image generator (Timgen) using npm (see startup command of
timgen
docker compose service) - Layman client using npm (see startup command of
layman_client
docker compose service) - Layman using your deployment server (see startup command of
layman
docker compose service) - Layman celery worker using python (see startup command of
celery_worker
docker compose service)
Suitable for development only.
Before the first run:
# use dev settings
cp .env.dev .env
Now everything is ready to start:
# start all needed dockerized containers
make start-dev
Initial startup may take few minutes (download docker images, build it, run it). Wait until you see something like
[2020-12-13 09:57:00,968] INFO in __init__: Layman successfully started!
in log of layman_dev
container:
# see logs from Layman
docker logs -f layman_dev
Then visit http://localhost:8000/. You will see simple web client that interacts with REST API. You can also log in with automatically provided Wagtail user layman
and password laymanpwd
.
To stop running service run:
# stop all needed dockerized containers
make stop-dev
By default, docker run all containers as root user. It's possible to change it by defining UID_GID
permanent environment variable. First stop all running containers:
make stop-dev
The UID_GID
variable should look like "<user id>:<group id>"
of current user, e.g. UID_GID="1000:1000"
. As it should be permanent, you can solve it for example by adding following line to your ~/.bashrc
file:
export UID_GID="1000:1000"
and restart terminal. Verification:
$ echo $UID_GID
1000:1000
Then change ownership of some directories
make prepare-dirs
sudo chown -R 1000:1000 layman_data/
sudo chown -R 1000:1000 layman_data_test/
sudo chown -R 1000:1000 src/
sudo chown -R 1000:1000 tmp/
and restart layman
make start-dev
- all files within LAYMAN_DATA_DIR!
- all files within LAYMAN_QGIS_DATA_DIR!
- all files within LAYMAN_GS_NORMALIZED_RASTER_DIRECTORY!
- all layman-related schemas in LAYMAN_PG_DBNAME!
- database EXTERNAL_DB_NAME and user READ_ONLY_USER
- all workspaces and Layman users in GeoServer!
- all keys in Redis logical database identified by LAYMAN_REDIS_URL!
- all keys in Redis logical database identified by LTC_REDIS_URL!
- metadata records from CSW identified by CSW_URL whose at least one online distribution URL contains LAYMAN_PROXY_SERVER_NAME!
Default values are defined in .env.test
# test related mainly to REST API endpoints of layers and maps
make test-static
make test-separated
# other tests
make test
For more information about tests, look at test-related documentation for developers.