Adding aws-s3 metric for sqs worker utilization

[kgeller]

What does this PR do?

Adding the metric to answer how many utilized the SQS workers are . 0 indicates free, 1 indicates all are working the entire polling duration of 5 seconds.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

http://localhost:5066/dataset?pretty now includes the following

"sqs_worker_utilization": 0.061649393

TODO do we want to limit how many decimal places?

Related issues

• Closes https://github.com/elastic/security-team/issues/5526

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b s3-metric-utilization upstream/s3-metric-utilization
git merge upstream/main
git push upstream s3-metric-utilization

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-04-01T16:18:26.496+0000

• Duration: 86 min 0 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2881
Skipped	179
Total	3060

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b s3-metric-utilization upstream/s3-metric-utilization
git merge upstream/main
git push upstream s3-metric-utilization

[andrewkroh]

I left some minor comments, but this structure looks good.

I do think there could be some inaccuracies with the metric value for long running SQS workers. For example if all five workers take 10 seconds to complete then in the first period the utilization will be 0 and then next it would be 2 (which is out of range).

[kgeller]

I do think there could be some inaccuracies with the metric value for long running SQS workers. For example if all five workers take 10 seconds to complete then in the first period the utilization will be 0 and then next it would be 2 (which is out of range).

@andrewkroh I added some more logic into the calculation function to resolve that situation.

[andrewkroh]

I think this would need to track the start times of each SQS worker to know if it was running for the whole 5 second interval or a fraction of it. I experimented with a change that uses a beginSQSWorker() and endSQSWorker() method to track each worker. I'm going to push that up and see what you think.

[andrewkroh]

What if we have two workers operating? One is long running (taking more than 5 seconds) and one is operating on short jobs (like completing every 100ms).

[andrewkroh]

/test

[kgeller]

I think this would need to track the start times of each SQS worker to know if it was running for the whole 5 second interval or a fraction of it. I experimented with a change that uses a beginSQSWorker() and endSQSWorker() method to track each worker. I'm going to push that up and see what you think.

I think those functions make sense. It seems much more thorough, while still being easy to follow