Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create an endpoint to trigger asset summary generation #391

Open
waxlamp opened this issue Jul 6, 2021 · 7 comments
Open

Create an endpoint to trigger asset summary generation #391

waxlamp opened this issue Jul 6, 2021 · 7 comments
Labels
DX Affects developer experience enhancement New feature or request

Comments

@waxlamp
Copy link
Member

waxlamp commented Jul 6, 2021

Yarik wrote:

@dchiquito do you see an easy quick way to trigger that offband generation of summaries for all dandisets?
may be we should expose some /updateInternal endpoint for each dandiset to facilitate such/similar updates in the future?

Originally posted by @yarikoptic in #389 (comment)

I would recommend the endpoint PUT /dandisets/{versions__dandiset__pk}/versions/{versions__version}/assets/summary for this purpose.

Design doc: https://github.com/dandi/dandi-api/blob/master/doc/design/scheduled-tasks.md

@satra
Copy link
Member

satra commented Jul 6, 2021

since the summary is at the dandiset level, perhaps:

/dandisets/{versions__dandiset__pk}/versions/{versions__version}/summary

note that at present any update to metadata should also trigger summarization, perhaps the summary endpoint is better, but it does also need to be triggered on a delay timer (relative to last asset update; preferred) or a cron-job (fallback).

however, more generally, we should consider how much of this should be in the api or part of the server workflow.

@yarikoptic
Copy link
Member

it does also need to be triggered on a delay timer (relative to last asset update; preferred)

Back ref #385

@dchiquito
Copy link
Contributor

Doing a PUT ../versions/{version_id} that is a no-op will update assetsSummary.

What exactly was the use case for this endpoint? assetsSummary should be recomputed whenever changes are made.

@yarikoptic
Copy link
Member

relates to #431 as might be providing a workaround for testing if this end-point would cause the task without any delay

@waxlamp
Copy link
Member Author

waxlamp commented Dec 31, 2021

@dchiquito, what's the status on this issue?

@dchiquito
Copy link
Contributor

I can see this being useful as way to quickly flush the asset summary queue when running tests. Right now the test has to wait at max 60s for the job to run, which is never a nice thing to do in a test.
This would be a nice optimization for the CLI tests, but it is a non-blocker.

@waxlamp waxlamp added DX Affects developer experience enhancement New feature or request labels Jan 4, 2022
@dchiquito
Copy link
Contributor

It's possible it might resolve the flaky test issues we've been seeing with asset summary validation #567 (comment)
Perhaps it's worth looking at sooner rather than later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DX Affects developer experience enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants