Marquez 0.42.0

Added

• Client: add Java client method for dataset/job lineage #2623 @davidjgoss
  To add a method for the dataset/job-level endpoint (GET /lineage) to the Java SDK, this adds a new method to the MarquezClient for the endpoint, along with tests, and the necessary new subclasses of NodeData for datasets and jobs.
• Web: add IO tab #2613 @phixme
  Improves experience with large graphs by adding a new tab to move between graph elements without looking at the graph itself.
• Web: add hover-over Tag tooltip to datasets #2630 @davidsharp7
  For parity with columns in the GUI, this adds a Tag tooltip to datasets.

Changed

• Docker: upgrade to Docker Compose V2 #2644 @merobi-hub
  Docker Compose V1 has been at EOL since June, but docker/up.sh uses the V1 format. This upgrades the up command in up.sh to V2.

Removed

• API: drop table job_contexts and usage #2621 @wslulciuc
  Removes usage of job_contexts, which has been replaced by OpenLineage facets, and adds a migration to drop the table.
• API: remove usage of current_job_context_uuid column #2622 @wslulciuc
  Removes usage of job_context_uuid and current_job_context_uuid. Column to be removed in 0.43.0.

Fixed

• Web: fix Unix epoch time display for null endedAt values #2647 @merobi-hub
  Fixes the issue of the GUI displaying Unix epoch time (midnight on January 1, 1970) in the case of running jobs/null endedAt values.

Marquez 0.41.0

Added

• API: add support for the following parameters in the SearchDao #2556 @tati @wslulciuc
  This PR updates the search endpoint to enforce YYYY-MM-DD for query params, use YYYY-MM-DD as LocalDate, and support the following query params:
  • namespace - matches jobs or datasets within the given namespace.
  • before - matches jobs or datasets before YYYY-MM-DD.
  • after - matches jobs or datasets after YYYY-MM-DD.
• Web: add paging on jobs and datasets #2614 @phixme
  Adds paging to jobs and datasets just like we already have on the lineage events page.
• Web: add tag descriptions to tooltips #2612 @davidsharp7
  Get the tag descriptions from the tags endpoint and when a column has a tag display the corresponding description on hover over. Context can be found here.
• Web: add available column-level tags #2606 @davidsharp7
  Adds a new column called "tags" to the dataset column view along with the tags associated with the dataset column.
• Web: add HTML Tool Tip #2601 @davidsharp7
  Adds a Tool Tip to display basic node details.

Fixed

• Web: fix dataset saga for paging #2615 @phixme
  Updates the saga, changes the default page size.
• API: perf/improve jobdao query #2609 @algorithmy1
  Optimizes the query to make use of Common Table Expressions to fetch the required data more efficiently and before the join, fixing a significant bottleneck.

Changed

• Docker: Postgres 14 #2607 @wslulciuc
  Bumps the recommended version of Postgres to 14.
  When deploying locally, you might need to run ./docker/down.sh to clean existing volumes.

Removed

• Client: tolerate null transformation attrs in field model #2600 @davidjgoss
  Removes the @NonNull annotation from the client class and the @NotNull from the model class.

Marquez 0.40.0

Added

• API: lineage events paging update #2577 @phixme
  Updates the API for lineage events and restyles the lineage events page to fix a number of bugs and code duplication.
• Chart: do not use hardcoded Postgres image for init container #2579 @terrpan
  Adds a template in chart/templates/helpers to use the global.imageRegistry input value for the wait-for-db container to improve performance on private registries.
• Web: add copy button for lineage ID #2578 @AmandaYao00
  Adds a copy button to the IDs on the Events page.

Fixed

• API: add defaults for idFromValue() and idFromValueAndType() #2581 @wslulciuc
  Replaces the null values in these functions in EventTypeResolver with defaults.
• Client: correct example syntax #2575 @davidjgoss
  Removes errant parens from the sample code's client instantiation.

Marquez 0.39.0

Added

• Web: add full graph toggle #2569 @jlukenoff
  Adds a toggle to the Lineage UI to let users switch between viewing the full graph and only the selected paths.
• Web: add ARIA labels to input fields #2562 @merobi-hub
  Adds i18next-compliant ARIA labels to input fields for improved accessibility.

Changed

• Web: upgrade React to version 18 #2563 @Xavier-Cliquennois
  Upgrades the Web client in order to utilize the latest version of Node.js and update all dependencies to their respective latest versions.

Fixed

• Web: fix the stylesheet for the date selector #2573 @phixme
  Fixes margins and moves the label to be more inline with what the defaults are to fix issues caused by the recent Material-UI upgrade.
• Web: update i18n for general search filter and runInfo facets search #2557 @merobi-hub
  Adds missing i18n support for runInfo and search.
• Docker: update web proxy import #2571 @phixme
  Updates the import style for the http-proxy-middleware.

Marquez 0.38.0

Added

• API: add db retention support #2486 @wslulciuc
  Adds migration, a dbRetention config in marquez.yml for enabling a retention policy, and a db-retention command for executing a policy.
• API: add runs state indices #2535 @phixme
  Adds four indices to help run retention faster.
• API: define DbRetentionJob(Jdbi, DbRetentionConfig) #2549 @wslulciuc
  Adds @Positive to DbRetentionConfig instance variables for validating DbRetentionConfig properties internally within the class.
• API: add log for when retention job starts #2551 @wslulciuc
  Adds logging of DbRetentionJob.

Fixed

• API: fix slow dataset query updates #2534 @phixme
  Scopes down nested facet queries to be the same scope as the outer query.
• Client/Python: increase namespace length to 1024 characters #2554 @hloomupgrade
  Changes the namespace length constraint to sync up with the Java client's.
• Web: remove pagination in case of no content #2559 @Nisarg-Chokshi
  Updates Dataset & Event route rendering to remove pagination in the case of no content.

Marquez 0.37.0

Added

• API: add ability to decode static metadata events #2495 @pawel-big-lebowski
  Introduces an EventTypeResolver for using the schemaURL field to decode POST requests to /lineage with LineageEvents, DatasetEvents or JobEvents, as the first step in implementing static lineage support.

Fixed

• API: remove unnecessary DB updates #2531 @pawel-big-lebowski
  Prevent updates that are not needed and are deadlock-prone.
• Web: revert URL encoding when fetching lineage #2529 @jlukenoff
  Reverts the node ID from being URL-encoded and allows the backend to return lineage details successfully even when a node ID contains special characters.

Marquez 0.36.0

Added

• UI: add an option for configuring the depth of the lineage graph #2525 @jlukenoff
  Makes the lineage UI a bit easier to navigate, especially for larger lineage graphs.

Fixed

• Docker: generate new uuid for etl_menus in seed data #2519 @wslulciuc
  Fixes a runID collision creating an invalid lineage graph when the seed command is used.
• Docker: remove unnecessary copy command from Dockerfile #2516 @Nisarg-Chokshi
  Deletes redundant copy command.
• Chart: enable RFC7230_LEGACY http compliance on application connectors by default #2524 @jlukenoff
  Adds this configuration to the helm chart by default to fix basic chart installation and ensure that the fix in #1419 does not revert.

Marquez 0.35.0

Added

• UI: add pagination to datasets #2512 @merobi-hub
  Adds pagination to the datasets route using the same approach employed for events.

Fixed

• UI: handle lineage graph cycles on the client #2506 @jlukenoff
  Fixes a bug where we blow the stack on the client-side if the user selects a node that is part of a cycle in the graph.

Marquez 0.34.0

Fixed

• Chart: skip regex after postgresql in chart/values.yaml #2488 @wslulciuc
  Fixes regex for version bump of chart/values.yaml in new-version.sh.

Marquez 0.33.0

Added

• API: support inputFacets and outputFacets from Openlineage specification #2417 @pawel-big-lebowski
  Adds the ability to store inputFacets / outputFacets sent within datasets, exposing them through the Marquez API as part of the Run resource.

Fixed

• API: fix job update SQL to correctly use simple_name for job updates #2457 collado-mike
  Fixes a bug in the job update logic stemming from use of the FQN rather than the simple_name and updates the relevant test.
• API: update SQL in backfill script for facet tables to improve performance #2461 collado-mike
  Dramatically improves migration performance by making the backfill script fetch events by run_uuid via a new temp table for tracking and sorting runs.
• API: update v61 migration to handle duplicate job names before unique constraint #2464 collado-mike
  To fix a bug in the case of duplicate job FQNs, this renames jobs that have been symlinked to point to newer versions of themselves so that the job FQN doesn't conflict and the unique constraint (without regard to parent job) can be applied. Note: Any installations that have already applied this migration will not see any new operations on their data, but installations that have duplicates will need this fix for the migration to complete successfully.
• API: make improvements to lineage query performance #2472 collado-mike
  Dramatically lessens the lineage query performance regression caused by removal of the jobs_fqn table in #2448.
• UI: change color for selected node and edges on graph #2458 tito12
  Improves the visibility of the selected node and edges by increasing the contrast with the background.
• UI: change color for selected node and edges on graph #2458 tito12
  Improves the visibility of the selected node and edges by increasing the contrast with the background.
• UI: Handle null run.jobVersion in DatasetInfo.tsx to fix rendering issues. [#2471] (#2471) perttus
  In some cases Marquez UI fails to render DatasetInfo, this addresses that issue.
• UI: better handling of null job latestRun for Jobs page #2467 perttus
  Fixes a bug where Jobs view fails to load where some jobs don't have latestRun.