This repository is an custom OpenMetadata Connector for GIS data formats.
We'll need to package the code so that it can be shipped to the ingestion container and used there. You can find a simple setup.py
that builds the connector
module.
If you want to use the connector from the UI, the openmetadata-ingestion
image should be aware of your new package.
We will be running the against the OpenMetadata version 1.2.0
, therefore, our Dockerfile looks like:
# Base image from the right version
FROM openmetadata/ingestion:1.2.0
# Let's use the same workdir as the ingestion image
WORKDIR ingestion
USER airflow
# Install our custom connector
# For a PROD image, this could be picking up the package from your private package index
COPY connector connector
COPY setup.py .
RUN pip install fiona
RUN pip install --no-deps .
Build and use the new openmetadata-ingestion images in Docker compose:
ingestion:
container_name: openmetadata_ingestion
build:
context: ../
dockerfile: docker/Dockerfile
We have a Makefile
prepared for you to run make run
. This will get OpenMetadata up in Docker Compose using the custom Ingestion image.
You may also just run:
docker compose -f ./docker/docker-compose.yml up -d
In this guide we prepared a Database Connector. Thus, go to Database Services > Add New Service > Custom
and set the Source Python Class Name
as connector.spatial_connector.SpatialConnector
.
Note how we are specifying the full module name so that the Ingestion Framework can import the Source class.
To run the OpenMetadata Spatial Connector, the Python class will be connector.spatial_connector.SpatialConnector
and we'll need to set the following Connection Options:
business_unit
: Any name you'd like (Optional - preferably no special characters).file_source
: The path or URL to the.shp/.gpkg
file. The connector uses Fiona which supports GDAL VFS.remote
:true
iffile_source
is a vfs url, e.g.zip+https://raw.githubusercontent.com/OSGeo/gdal/master/autotest/ogr/data/shp/poly.zip
or/vsizip//vsicurl/https://raw.githubusercontent.com/OSGeo/gdal/master/autotest/ogr/data/shp/poly.zip
. Can be omitted otherwise.
Everyone is invited to get involved and contribute to the project.
Simply create a fork and pull request for code contributions or feel free to open an issue for any other contributions or issues.