Schedule copay notification jobs in time intervals

[ScottyRJames]

Summary

• Sidekiq limiter was ideal with our use case
  • Window/bucket caused jobs to fail
  • Concurrent blocked other jobs from being processed
• Use throttle approach
  • Schedule a certain amount of jobs for every time period
  • Period and batch size determined by settings values
  • Tweak as needed

Related issue(s)

- department-of-veterans-affairs/va.gov-team#0000
https://github.com/department-of-veterans-affairs/devops/pull/13134
https://github.com/department-of-veterans-affairs/vsp-infra-application-manifests/pull/2010

Testing done

• Tested with local sidekiq queue

What areas of the site does it impact?

Sidekiq/copay notifications

Acceptance criteria

• I fixed|updated|added unit tests and integration tests for each feature (if applicable).
• No error nor warning in the console.
• Events are being sent to the appropriate logging solution
• Documentation has been updated (link to documentation)
• No sensitive information (i.e. PII/credentials/internal URLs/etc.) is captured in logging, hardcoded, or specs
• Feature/bug has a monitor built into Datadog or Grafana (if applicable)
• If app impacted requires authentication, did you login to a local build and verify all authenticated routes work as expected
• I added a screenshot of the developed feature

Requested Feedback

Any concern for spinning up batches of jobs every couple minutes?

[ryan-mcneil]

@ScottyRJames just curious - do you have other team members that you could get to give their blessing before it gets to us? It's a process that we're slowly rolling out/requesting for all PRs, as it's difficult for us to give thorough PR reviews, especially when we don't have the full context or understanding of the business reqs.

[kjsuarez]

LGTM 👍

[ScottyRJames]

@ryan-mcneil Thanks for the heads up on this new process, @kjsuarez is another BE engineer on the debt resolution team and left a review earlier. Also added these files to our team within CODEOWNERSHIP to try to reduce the platform dependency for our team's managed code

[ryan-mcneil]

@ScottyRJames I just wanted to make sure this is what you intended. If JOB_INTERVAL and BATCH_SIZE were 90 and 100 respectively, then we'd be kicking off a new job every .9 seconds (not JOB_INTERVAL seconds), correct? Maybe I'm missing something?

[ScottyRJames]

@ryan-mcneil It will kick off 100 new jobs every 90 seconds rather since ruby division defaults to the floor with integers.

For example, if one of the statements had index 127 it would be evaluated like 90 * (127 / 100) -> 90 * 1 which perform_in will take and schedule the job at 90 seconds later. So if unique_statements were to be 500 statements long each of the slated times would be like

Indices	Start Time
0 - 99	Time.now
100 - 199	Time.now + 90s
200 - 299	Time.now + 180s
300 - 399	Time.now + 270s
400 - 499	Time.now + 360s
500	Time.now + 450s

If we wanted to go towards your thinking, it would look like like JOB_INTERVAL * (index / BATCH_SIZE).to_f. I think by default Sidekiq checks for new jobs every 5 seconds so it wouldn't be to the exact millisecond that it's scheduled at. It's still workable if you think it would be better to stagger the jobs even more rather than spinning up a batch at a time but it would be more like 6-7 new jobs every 5 seconds vs a new job every .9 seconds.

[ryan-mcneil]

Yeah, you're right. I wasn't considering the ruby division. Just wanted to make sure you considered it, and it looks like you're a step ahead of me 👌

[HeatherRienks]

following

[ryan-mcneil]

👌