Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CCP] - Cron jobs need to have randomized times #731

Closed
gitAskur opened this issue Dec 18, 2020 · 2 comments · Fixed by eveseat/console#87 or eveseat/console#88
Closed

[CCP] - Cron jobs need to have randomized times #731

gitAskur opened this issue Dec 18, 2020 · 2 comments · Fixed by eveseat/console#87 or eveseat/console#88
Labels
esi

Comments

@gitAskur
Copy link

gitAskur commented Dec 18, 2020 •
edited
Loading

  • Problem: SeAT is configured to run cronjobs at the same time everywhere, resulting in all instances hitting our servers at the same time causing load spikes. These spikes will grow and grow as SeAT is adopted by more people and is untenable.
  • Expected: Spread out a randomized intervals.

Hit me up on Slack for any clarification.

If this can not be achieved we will have to block the user agent.
@warlof warlof added the esi label Dec 18, 2020
@warlof warlof pinned this issue Dec 18, 2020
@warlof
Copy link
Member

warlof commented Dec 18, 2020 •
edited
Loading

some hint

warlof added a commit to warlof/eveseat-console that referenced this issue Dec 18, 2020
@warlof
feat: delay queued jobs using a random window
79fab3e 
to prevent excessive call at time from all existing SeAT instances in the world, add some randomness to token update batches.

also queue jobs related to corporation only for token having a Director role.
most of corporation endpoints are requiring that role and that should reduce the amount of calls by the meantime.

seed scheduler using random values overriding default scheduler at every time it's ran.
to ensure all existing instance benefits of the change, we force scheduler to be seeded even if commands already exist.

EVE Online maintenance window is excluded from schedule - still need to find a way to handle timezone though :(

Closes eveseat/seat#731
@warlof warlof mentioned this issue Dec 18, 2020
Merged
warlof added a commit to eveseat/console that referenced this issue Dec 21, 2020
@warlof
feat: delay queued jobs using a random window (#87)
a5ae475 
to prevent excessive call at time from all existing SeAT instances in the world, add some randomness to token update batches.

also queue jobs related to corporation only for token having a Director role.
most of corporation endpoints are requiring that role and that should reduce the amount of calls by the meantime.

seed scheduler using random values overriding default scheduler at every time it's ran.
to ensure all existing instance benefits of the change, we force scheduler to be seeded even if commands already exist.

Related to eveseat/seat#731
@warlof warlof reopened this Dec 21, 2020
@warlof
Copy link
Member

warlof commented Dec 21, 2020 •
edited
Loading

I leave the issue open for now as we attempt to design a new way to handle updates related to this issue which should help to reduce the load across a timeframe in a more efficient way. However, this will not be done until next year.

Instances with agent Eseye/2.3.0/SeAT 4.7.1 or greater are using randomizer to manage jobs -> https://github.com/eveseat/console/releases/tag/4.6.0

@warlof warlof mentioned this issue Dec 27, 2020
Merged
3 tasks
@warlof warlof unpinned this issue Jan 3, 2021
@Crypta-Eve Crypta-Eve mentioned this issue Nov 21, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
esi
Projects
None yet
Milestone
No milestone
2 participants
@warlof @gitAskur