-
Notifications
You must be signed in to change notification settings - Fork 143
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace slow json filename resolver #306
Replace slow json filename resolver #306
Conversation
10d72f2
to
7745952
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice find
Looks like this repo's CI needs to be migrated to github |
def test_pkgutil_static_read(benchmark): | ||
def get_sampling_rule(): | ||
json.loads(pkgutil.get_data(__name__, 'mock_sampling_rule.json')) | ||
benchmark(get_sampling_rule) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These tests are really testing the performance of standard library functions (pkgutil
and pkg_resources
) instead of testing aws-xray-sdk-python
functions.
For that reason I think we should not include them here? The PR comment can serve as proof that this change is worth it.
Otherwise, we can leave the tests here so the results can be reproduced later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree with deleting these benchmarks (first thing that came to mind but keeping them also isn't a big deal)
Noticed this after reading getmoto/moto#4142 Ref: aws/aws-xray-sdk-python#306 git-svn-id: file:///srv/repos/svn-community/svn@1003952 9fca08f4-af9d-4005-b8df-a31f2cc04f65
Noticed this after reading getmoto/moto#4142 Ref: aws/aws-xray-sdk-python#306 git-svn-id: file:///srv/repos/svn-community/svn@1003952 9fca08f4-af9d-4005-b8df-a31f2cc04f65
Bumps [aws-xray-sdk](https://github.com/aws/aws-xray-sdk-python) from 2.8.0 to 2.9.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/aws/aws-xray-sdk-python/releases">aws-xray-sdk's releases</a>.</em></p> <blockquote> <h2>2.9.0 Release</h2> <p>See details in <a href="https://github.com/aws/aws-xray-sdk-python/blob/master/CHANGELOG.rst">CHANGELOG</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/aws/aws-xray-sdk-python/blob/master/CHANGELOG.rst">aws-xray-sdk's changelog</a>.</em></p> <blockquote> <h1>2.9.0</h1> <ul> <li>bugfix: Change logging behavior to avoid overflow. <code>PR302 <https://github.com/aws/aws-xray-sdk-python/pull/302></code>_.</li> <li>improvement: Lazy load samplers to speed up cold start in lambda. <code>PR312 <https://github.com/aws/aws-xray-sdk-python/pull/312></code>_.</li> <li>improvement: Replace slow json file name resolver. <code>PR 306 <https://github.com/aws/aws-xray-sdk-python/pull/306></code>_.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/aws/aws-xray-sdk-python/commit/be4dea2f89e44dd52c68b23e1620b8541947e64b"><code>be4dea2</code></a> Release commit for v2.9.0 (<a href="https://github-redirect.dependabot.com/aws/aws-xray-sdk-python/issues/318">#318</a>)</li> <li><a href="https://github.com/aws/aws-xray-sdk-python/commit/9858cab26dd87b26cfbec852449cfcfc9be4d73c"><code>9858cab</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/aws/aws-xray-sdk-python/issues/317">#317</a> from wangzlei/master</li> <li><a href="https://github.com/aws/aws-xray-sdk-python/commit/85d8801d7b00ce3b97ede94e3181db5ebe7ad09d"><code>85d8801</code></a> Remove redundant error log MISSING_SEGMENT_MSG</li> <li><a href="https://github.com/aws/aws-xray-sdk-python/commit/05f5e8f7ab7b17c5f9f80ac01c97712a8737ceed"><code>05f5e8f</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/aws/aws-xray-sdk-python/issues/315">#315</a> from aws/willarmiros-patch-1</li> <li><a href="https://github.com/aws/aws-xray-sdk-python/commit/e1841e668a46b0da7b83e5fdf0bb231cedf2dfb3"><code>e1841e6</code></a> Create CODEOWNERS</li> <li><a href="https://github.com/aws/aws-xray-sdk-python/commit/0e1f935bd2040ee7dbf0625db7f7ad780c66fb37"><code>0e1f935</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/aws/aws-xray-sdk-python/issues/312">#312</a> from maxday/maxday/lazy-load-samplers</li> <li><a href="https://github.com/aws/aws-xray-sdk-python/commit/f4b33f0920a28d87a59979e1262b5511c80a6dfd"><code>f4b33f0</code></a> lazy load samplers</li> <li><a href="https://github.com/aws/aws-xray-sdk-python/commit/86248a528a43a1cab2c5ff50056eb01f3ff8d8a6"><code>86248a5</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/aws/aws-xray-sdk-python/issues/306">#306</a> from NathanielRN/remove-slow-filepath-resolve</li> <li><a href="https://github.com/aws/aws-xray-sdk-python/commit/aa4b2f5827bcfdf6039753cd547fcb34611684c9"><code>aa4b2f5</code></a> Benchmark tests should both consistently return</li> <li><a href="https://github.com/aws/aws-xray-sdk-python/commit/8ad460eb5a6de440939f5ce22cfc01ac7cca7613"><code>8ad460e</code></a> Add benchmarks for json read</li> <li>Additional commits viewable in <a href="https://github.com/aws/aws-xray-sdk-python/compare/2.8.0...2.9.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=aws-xray-sdk&package-manager=pip&previous-version=2.8.0&new-version=2.9.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>
Issue #305
Description of changes:
For optimization purposes, we found that
pkgutil.get_data(package, resource)
method could be used to read a file in a much faster way thanfrom pkg_resources import resource_filename
.Learn more about reading static files in Python.
How did I test
I used
pytest-benchmark
to compare the performance of the two:which shows that
pkgutil
is faster.Noteworthy points
pkgutil
is standard inpython2
as well, so it should work fine with it.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.