Broken LineageGraph in 0.43.0

[wslulciuc]

Marquez 0.43.0 introduced a regression in our lineage graph (broken lineage) as a result in an update to our lineage query.

[wslulciuc]

The error in the graph might be seed data related. Using the calls from the OpenLineage getting started guide, we see the lineage graph and the current version set as TRUE in the I/O mapping table:

[wslulciuc]

@pawel-big-lebowski, after troubleshooting the error more, it looks like the lineage graph "breaks" when OL events are sent twice. That is, sending the followings events once, produces the correct lineage graph (as mentioned above):

# (1) Send START event
curl -X POST http://localhost:5004/api/v1/lineage \
  -i -H 'Content-Type: application/json' \
  -d '{
        "eventType": "START",
        "eventTime": "2020-12-28T19:52:00.001+10:00",
        "run": {
          "runId": "d46e465b-d358-4d32-83d4-df660ff614dd"
        },
        "job": {
          "namespace": "my-namespace",
          "name": "my-job"
        },
        "inputs": [{
          "namespace": "my-namespace",
          "name": "my-input"
        }],  
        "producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
        "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
      }'
# (2) Send COMPLETE event
curl -X POST http://localhost:5004/api/v1/lineage \
  -i -H 'Content-Type: application/json' \
  -d '{
        "eventType": "COMPLETE",
        "eventTime": "2020-12-28T20:52:00.001+10:00",
        "run": {
          "runId": "d46e465b-d358-4d32-83d4-df660ff614dd"
        },
        "job": {
          "namespace": "my-namespace",
          "name": "my-job"
        },
        "outputs": [{
          "namespace": "my-namespace",
          "name": "my-output",
          "facets": {
            "schema": {
              "_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
              "_schemaURL": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/spec/OpenLineage.json#/definitions/SchemaDatasetFacet",
              "fields": [
                { "name": "a", "type": "VARCHAR"},
                { "name": "b", "type": "VARCHAR"}
              ]
            }
          }
        }],
        "producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
        "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
      }'

But, if we send the events above for the 2nd time, the lineage graph is no longer connected:

I think the bug has to do with the markInputOrOutputDatasetAsPreviousFor() methods in JobVersionDao