Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate namespace to be optional in API and WEB #2876

Open
phixMe opened this issue Aug 20, 2024 · 1 comment
Open

Migrate namespace to be optional in API and WEB #2876

phixMe opened this issue Aug 20, 2024 · 1 comment
Labels
api API layer changes feature help wanted Extra attention is needed proposal web
Milestone

Comments

@phixMe
Copy link
Member

phixMe commented Aug 20, 2024

Currently, the endpoints for marquez all look along the lines of:
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}

This means that namespaces are effectively mandatory when users are working with the api and thus the web contains a dropdown allowing users to select a namespace. This is not ideal for a few reasons:

  1. We either need to default a namespace (1st in the list) or force a user to select a namespace before anything else can be rendered.
  2. A user does not always know the namespace of a job or dataset they are looking for because many of the OL integrations (Spark, Airflow, etc...) assign these.
  3. A long list of namespaces is not going to be ideal since a MQ user with thousands of namespaces will need to scroll a long way and have application rendering degraded.

Therefore, we need to handle namespaces a little differently across the board to rectify these problems listed above.

Proposal

Migrate the format of the endpoints to use query parameters like below and then forward old formats to new formats for a few releases (or forever)

# OLD
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}
# NEW
http://localhost:5000/api/v1/datasets/{dataset}?namespace={namespace}

Update the web to just render all the jobs and datasets, paginated of course without regard to namespace. Include a namespace filtering mechanism with a typeahead select so that users with many namespaces can find there known namespaces and filter.

@phixMe phixMe added help wanted Extra attention is needed feature api API layer changes proposal web labels Aug 20, 2024
@wslulciuc wslulciuc added this to the 0.51.0 milestone Oct 23, 2024
@wslulciuc
Copy link
Member

wslulciuc commented Oct 23, 2024

0.50.0 release will allow for listing jobs without a namespace via GET /api/v1/jobs, see #2930. We will be taking a similar approach for datasets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api API layer changes feature help wanted Extra attention is needed proposal web
Projects
Status: No status
Development

No branches or pull requests

2 participants