Skip to content
This repository has been archived by the owner on Jun 1, 2024. It is now read-only.

takezoe/trino-openlineage

Repository files navigation

Caution

This repository is no longer maintaned since OpenLineage integration has been merged into Trino.

trino-openlineage

An experimental OpenLineage integration for Trino

Marquez

Requirements

  • Java 21
  • Maven
  • Trino 422

Mapping of properties

OpenLineage Trino
JobFacet Name QueryID
JobFacet Namespace Query Environment (or configured in event listener)
RunFacet ID QueryID

Installation

Build and copy trino-openlineage plugin:

mvn clean install -DskipTests
unzip ./target/trino-openlineage-422.zip -d $TRINO_HOME/plugin

Add the following line to $TRINO_HOME/etc/event-listener.properties:

event-listener.name=openlineage
openlineage.url=http://localhost:5000
#openlineage.facets.trinoMetadata.enabled=false
#openlineage.facets.trinoQueryContext.enabled=true
#openlineage.facets.trinoQueryStatistics.enabled=false
#openlineage.namespace=default
#openlineage.apikey=xxxx

Local testing

  1. Build plugin:
mvn clean install -DskipTests
  1. Run docker compose:
docker compose up -d
  • Freshly built plugin will be automatically mounted to your trino pod.
  • Configuration of the plugin will be taken from event-listener.properties file - adjust it to your will and restart trino pod for changes to take effect.
  1. Run query creating new table:
docker exec -it oltrino trino --execute 'create table memory.default.test_table as select * from tpch.sf1.nation limit 1;'
  1. Check logs of mock api:
docker logs olapi

About

OpenLineage integration for Trino

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages