Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Downloads may block API #65

Closed
nikromen opened this issue Jan 7, 2024 · 1 comment
Closed

Downloads may block API #65

nikromen opened this issue Jan 7, 2024 · 1 comment

Comments

@nikromen
Copy link
Member

nikromen commented Jan 7, 2024

The download takes some time, we have 8 workers - 8 downloads close to each other (but I don't think it will be so demanding feature) -> API blocked for a while.

If this will sometimes happen:

  • make the download endpoint asynchronous
  • switch to uvicorn from gunicorn (uvicorn is asynchronous server, gunicorn not)
@github-project-automation github-project-automation bot moved this to Needs triage in CPT Kanban Jan 7, 2024
@nikromen nikromen mentioned this issue Jan 7, 2024
Merged
nikromen added a commit that referenced this issue Jan 8, 2024
@nikromen
backend: create endpoint for downloading data created by cron
984825d 
This solution raises two concerns for the future once we have a
lot of data collected:
- it will be a while until the data will be tarred - creating delay before
  actual download (after cca 200 feedbacks relly noticable delay :/)
  look at example with only 300 (tiny!!) feedbacks:
  [root@backend persistent]# time tar -zcf results.tar.gz results/
  real m9.473s
  user m9.328s
  sys m0.483s
- downloading takes also some time

-> thus blocking the whole worker during this. IIRC we have 8 workers thus
8 downloads and API is unresponsive.

Solution:
- how to solve the delay before download:
  #64
- the issue above is the slowest, once that will be resolved and people
  still complain, do this:
  #65
@praiskup
Copy link
Member

We can set a limit per/route once the problem appears. Closing for now.

@praiskup praiskup moved this from Needs triage to Done in CPT Kanban Jan 10, 2024
nikromen added a commit to nikromen/log-detective-website that referenced this issue Jul 29, 2024
@nikromen
backend: create endpoint for downloading data created by cron
e42370b 
This solution raises two concerns for the future once we have a
lot of data collected:
- it will be a while until the data will be tarred - creating delay before
  actual download (after cca 200 feedbacks relly noticable delay :/)
  look at example with only 300 (tiny!!) feedbacks:
  [root@backend persistent]# time tar -zcf results.tar.gz results/
  real m9.473s
  user m9.328s
  sys m0.483s
- downloading takes also some time

-> thus blocking the whole worker during this. IIRC we have 8 workers thus
8 downloads and API is unresponsive.

Solution:
- how to solve the delay before download:
  fedora-copr#64
- the issue above is the slowest, once that will be resolved and people
  still complain, do this:
  fedora-copr#65
nikromen added a commit to nikromen/log-detective-website that referenced this issue Jul 29, 2024
@nikromen
backend: create endpoint for downloading data created by cron
93bda6f 
This solution raises two concerns for the future once we have a
lot of data collected:
- it will be a while until the data will be tarred - creating delay before
  actual download (after cca 200 feedbacks relly noticable delay :/)
  look at example with only 300 (tiny!!) feedbacks:
  [root@backend persistent]# time tar -zcf results.tar.gz results/
  real m9.473s
  user m9.328s
  sys m0.483s
- downloading takes also some time

-> thus blocking the whole worker during this. IIRC we have 8 workers thus
8 downloads and API is unresponsive.

Solution:
- how to solve the delay before download:
  fedora-copr#64
- the issue above is the slowest, once that will be resolved and people
  still complain, do this:
  fedora-copr#65
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
CPT Kanban
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@praiskup @nikromen