delete: add endpoint for deleting namespaces #2244
Conversation
mobuchowski
changed the title
Codecov Report
@@ Coverage Diff @@
## main #2244 +/- ##
=========================================
Coverage 76.72% 76.72%
Complexity 1147 1147
=========================================
Files 219 219
Lines 5318 5318
Branches 423 423
=========================================
Hits 4080 4080
Misses 763 763
Partials 475 475 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
mobuchowski
force-pushed
the
delete/namespace-endpoint
branch
from
3d324c7 to
603c147
Compare
|
|
||
| jobs = client.listJobs(namespaceName); | ||
| assertThat(jobs).hasSize(1); | ||
| } |
There was a problem hiding this comment.
Could we add some undelete scenario?
There was a problem hiding this comment.
The test does check the number of datasets after the namespace is undeleted. But, looks like that's a new dataset added to the namespace and not the original datasets?
There was a problem hiding this comment.
Yes, the idea here is not to automatically undelete everything in the namespace.
api/src/test/java/marquez/db/migrations/V44_2__BackfillAirflowParentRunsTest.java
Show resolved
Hide resolved
mobuchowski
force-pushed
the
delete/namespace-endpoint
branch
4 times, most recently
from
72b4ec7 to
cbcec83
Compare
| .findBy(name.getValue()) | ||
| .orElseThrow(() -> new NamespaceNotFoundException(name)); | ||
| datasetService.deleteByNamespaceName(namespace.getName().getValue()); | ||
| jobService.deleteByNamespaceName(namespace.getName().getValue()); |
There was a problem hiding this comment.
What if we soft delete the dataset and job, but an error happens right before we soft delete the namespace?
There was a problem hiding this comment.
Only thing would be that visually empty namespace would appear on the UI, as this is filtered on the frontend.
There was a problem hiding this comment.
Do you think we should have more resiliency about those type of failures? I think this operation is idempotent, as long as there are no followup events for the same namespace.
There was a problem hiding this comment.
I think it's fine as is. Perhaps I would have had NamespaceService.delete() soft delete the dataset and job within a transaction (in the DAO layer).
| status: | ||
| patch: off | ||
|
|
||
| ignore: | ||
| - "api/src/main/java/marquez/db/migrations/V44_1__UpdateRunsWithJobUUID.java" |
There was a problem hiding this comment.
OOh, cool! Had no idea this was even an option 💯
mobuchowski
force-pushed
the
delete/namespace-endpoint
branch
3 times, most recently
from
8cc6eb7 to
342a1c6
Compare
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
mobuchowski
force-pushed
the
delete/namespace-endpoint
branch
from
342a1c6 to
b882daa
Compare
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
mobuchowski
force-pushed
the
delete/namespace-endpoint
branch
from
b882daa to
74abc92
Compare
Add option to hide/soft delete namespace.
When namespace is hidden, all datasets and jobs contained by this namespace are also hidden. Namespace stops being shown on UI - frontend filters it by isHidden attribute, instead of filtering on backend. Reason for doing it this way is huge potential complexity of changing namespace contract - for example, a lot of methods check whether namespace exists before performing.
Deleted namespace is being undeleted when relevant OpenLineage event is received. This does not automatically undelete all the datasets and jobs in the namespace, only those that are received in this event.
Also, fixes bug where deleted child job (from an event with
ParentRunFacet) wasn't getting deleted due to internal job name handling.Closes: #2095
Signed-off-by: Maciej Obuchowski obuchowski.maciej@gmail.com