Skip to content

Latest commit

 

History

History
507 lines (439 loc) · 17.3 KB

README.md

File metadata and controls

507 lines (439 loc) · 17.3 KB
Raw

Smart MaaS
SMART MOBILITY SERVICE PLATFORM
Smart MaaS

Contact • Issues • Project Page

vbb-rest-2-ngsi

MIT license Support badge NGSI v2 NGSI-LD badge

Node.js application that uses the vbb-rest API to read public transport data of the Verkehrsverbund Berlin-Brandenburg (VBB) and transform it from FTPF into a NGSI-compliant data format (NGSI v2 and NGSI-LD) for subsequent storage in FIWARE NGSI Context Brokers. Applied FIWARE data models implement the General Transit Feed Specification (GTFS).

Content

Current limitations

  • The NGSI data model types GtfsService and GtfsTrip are not considered yet. Although with regard to the relationship model these types should be harmonised, it's due to the implementation of the REST API that some information is not available or suitable for transformation into context data.

    • A GtfsService should reference a GtfsAgency but various defined service times for routes are not known or cannot be queried via API.

    • A GtfsTrip may reference a GtfsShape, should reference a GtfsService and a GtfsRoute, but a trip ID always includes date of the day and is often regenerated. Therefore its form is not suitable for transformation into a NGSI entity with fixed ID.

  • Due to runtime reasons, instead of querying all stations (~ 13000) from the vbb-rest API only a static subset of 10 selected stations is processed for now.
    Furthermore just one possible journey of 10 selected connections (station A -> station B) is picked. It is planned to include all stops/stations and journeys at some point.

Prerequisites

Docker and Docker Compose must be installed on your system in order to run the services defined in the multi-container file.

Operation modes

The project offers two different compose files. The first variant starts the Node.js script, the context broker and components for persisting data. In this mode (client-server mode), the retrieved public transport data is stored in the context broker of the local Docker container and, if configured, persisted in the local CrateDB instance.

The second variant comprises a single service for the Node.js script. It acts as a client to a context broker running elsewhere (client mode).

Configuration

There is a configuration file .env containing environment variables used by the Node.js script. Some of the variables values have to be modified prior to initial startup, as the script uses those variables for connection management and data processing.

The following list gives a summary of currently supported variables and their description:

Name Description Default value
Application settings
APP_START_DELAY

delayed start: give other required services some time to get started before trying to connect to them [seconds]

mandatory

 20
VBB REST Data Source
VBBREST_BASE_URL

base URL (VBB REST data source)

mandatory

 https://3.vbb.transport.rest
VBBREST_QUERY_INTERVAL

interval for re-read data from REST API [seconds]

NOTE: 24 hours (86400 sec.) as default, please do not stress this public API too much!

mandatory

 86400
NGSI v2 Context Broker
BROKER_V2_BASE_URL

NGSI v2 context broker URL

NOTE: modify ONLY, when communicating with external context brokers, e.g. when executing './services-app-only' which does NOT start any context broker!
If this broker should not be used, just set an empty value.

optional

 http://orion-v2:1026/v2
BROKER_V2_AUTH_KEY

Auth key for 'Authorization' header

optional
BROKER_V2_API_KEY

API key (token for authenticaton)

optional
BROKER_V2_TENANT

tenant name (a tenant is a service aka domain on the context broker with its own isolated logical database)

optional

 vbb
BROKER_V2_SUBTENANT

sub-tenant name (a sub-tenant is a sub-service / service path aka project for the given tenant)

optional

 /public_transport
BROKER_V2_ENTITY_ID_SUFFIX

entity ID suffix (on creation will be appended to an entitys ID for a customized identification format, e.g. the ID suffix 'XY' for a GtfsStop entity '650030877901' will result in GtfsStop:650030877901:XY)

optional
NGSI-LD Context Broker
BROKER_LD_BASE_URL

NGSI-LD context broker URL

NOTE: modify ONLY, when communicating with external context brokers, e.g. when executing './services-app-only' which does NOT start any context broker!
If this broker should not be used, just set an empty value.

optional

 http://orion-ld:1026/ngsi-ld/v1
BROKER_LD_AUTH_KEY

Auth key for 'Authorization' header

optional
BROKER_LD_API_KEY

API key (token for authenticaton)

optional
BROKER_LD_TENANT

tenant name (a tenant is a service aka domain on the context broker with its own isolated logical database)

optional

 vbb
BROKER_LD_SUBTENANT

sub-tenant name (a sub-tenant is a sub-service / service path aka project for the given tenant)

optional

 /public_transport
BROKER_LD_ENTITY_ID_SUFFIX

entity ID suffix (on creation will be appended to an entitys ID for a customized identification format, e.g. the ID suffix 'XY' for a GtfsStop entity '650030877901' will result in urn:ngsi-ld:GtfsStop:650030877901:XY)

optional
Historic data persistence
ENABLE_HISTORIC_DATA_STORAGE

enables storage of historic data (into Crate-DB via QuantumLeap API for now) - support for NGSI v2 data only

optional

 true
QL_V2_NOTIFICATION_BASE_URL

QuantumLeap (QL) notification URL used for sending status changes of entities in the context broker

NOTE: modify ONLY, when communicating with external QL instances, e.g. when executing './services-app-only' which does NOT start any QL instance!
If historic data persistence via QL is not wanted, just set an empty value.

optional

 http://quantumleap:8668/v2
QL_V2_AUTH_KEY

Auth key for 'Authorization' header in requests to QL

optional
QL_V2_API_KEY

API key for authentication on QL

optional
QL_V2_TENANT

tenant name on QL

optional

 vbb
QL_V2_SUBTENANT

sub-tenant name on QL

optional

 /public_transport

Starting Docker containers

Depending on what operation mode is preferred, pull/create the images and start containers by running ./services create && ./services start (client-server mode) or simply ./services-app-only start (client mode) from the project root folder.
To stop the containers run ./services[-app-only] stop.
If you encounter problems executing the service scripts, add the missing permission with chmod +x services*.

GTFS entity relationships

GTFS entity relationships

NGSI data models

Read data from the vbb-rest API is harmonised for transformation into applicable FIWARE NGSI data models of the domain Urban Mobility.
Entites of the following types are sent to the context brokers:

  • GtfsStop - A stop where vehicles pick up or drop off riders
  • GtfsStation - A physical structure or area that contains one or more stops
  • GtfsAgency - Transit agency (transportation operator) that offers services at particular times
  • GtfsRoute - A (transit) route is a group of trips (sequences of two or more stops that occur during a specific time period) that are displayed to riders as a single service.
  • GtfsShape - Describes the path that a vehicle travels along a route alignment (consists of a sequence of points)

Entity IDs use the following format:

Entity Type Entity ID format
GtfsStop

GtfsStop:<STOP_ID>[:<ENTITY_ID_SUFFIX>]

e.g. GtfsStop:000008012713:XY
GtfsStation

GtfsStation:<STATION_ID>[:<ENTITY_ID_SUFFIX>]

e.g. GtfsStation:900000245025:XY
GtfsAgency

GtfsAgency:<AGENCY_ID>[:<ENTITY_ID_SUFFIX>]

e.g. GtfsAgency:berliner-verkehrsbetriebe:XY
GtfsRoute

GtfsRoute:<ORIGIN_STATION_ID>-<DESTINATION_STATION_ID>[:<ENTITY_ID_SUFFIX>]

e.g. GtfsRoute:900000044201-900000009202:XY
GtfsShape

GtfsShape:<ORIGIN_STATION_ID>-<DESTINATION_STATION_ID>[:<ENTITY_ID_SUFFIX>]

e.g. GtfsShape:900000044201-900000009202:XY

Notes:

  • <ENTITY_ID_SUFFIX> is an optional ID suffix that will be appended to each entity ID (preceded by a colon ':') if configured in the configuration.
  • The same rules apply to NGSI-LD entities. The only difference: urn:ngsi-ld: is prepended to the entity ID, e.g.

    urn:ngsi-ld:GtfsStop:000008012713:XY`

Reading data from context brokers

You can GET a list of all entities using the following cURL commands. Don't forget to replace the <DOCKER_HOST> placeholders with the hostname / IP of your Docker host.
<DOCKER_HOST> assumes that you are running your own context brokers on the Docker host. If you are connecting to context brokers located elsewhere, use their hostname / IP address accordingly.

Orion v2

List all GtfsStation entities containing only the 'id' attribute:

curl -X GET '<DOCKER_HOST>:1026/v2/entities?type=GtfsStation&attrs=id&options=keyValues' \
  -H 'Accept: application/json'

List all GtfsStation entities containing all attributes:

curl -X GET '<DOCKER_HOST>:1026/v2/entities?type=GtfsStation&options=keyValues' \
  -H 'Accept: application/json'

History data

If historic data persistence was enabled, the Node.js script sends subscriptions for attribute changes of all known entities to the NGSI v2 context broker. QuantumLeap, a REST service for storing, querying and retrieving spatial-temporal data, will receive a notification every time a status changes and stores its current value in a CrateDB database. With this data collected over time, statistical evaluations and data visualisation will be possible, e.g. building histograms with Grafana or UI widgets using WireCloud.

As QuantumLeap has no support for NGSI-LD yet, storage of historic data is supported for NGSI v2 data only.

The Docker container of CrateDB exposes 4200 as the default port for data queries and access to the web-based admin UI.
You can reach it at <DOCKER_HOST>:4200.

Troubleshooting

CrateDB service might crash shortly after startup due to incompatible memory settings.
With docker ps -a you can check whether its container is running or has already exited. In the latter case, inspecting the container log file with

sudo vi `docker inspect --format='{{.LogPath}}' kipark-db-crate`

should give you an output saying something like
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144].

In order to avoid this, increase maximum number of memory map areas before starting:

sudo sysctl -w vm.max_map_count=262144

Known issues

Due to some open issues of the NGSI-LD context broker, unexpected behaviour and errors may appear while executing the script.

License

This project is licensed under MIT License.