Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for non-mainnet request cache validation thresholds #3508

Merged
merged 6 commits into from
Oct 15, 2024

Conversation

fselmo
Copy link
Collaborator

@fselmo fselmo commented Oct 10, 2024

What was wrong?

closes #3506

How was it fixed?

  • If non-mainnet ethereum, allow the use of an integer value for the request cache validation threshold. Pre-configure "safe" default values for some common non-mainnet chain ids based on their varied finality mechanisms.
  • This integer value represents the number of seconds from time.now() that the request cache deems as a safe enough time window to allow the request to be cached.
  • Update the tests to reflect these changes.
  • Added comprehensive testing around the caching logic, affected endpoints, and all relevant providers.

Unrelated:

  • Add a note to the request_mocker to make sure that mocked results
    are of the correct expected types based on JSON-RPC spec (e.g. hex
    strings instead of ints for numbers, etc.)
  • Removed request caching decorator from base providers since they don't implement make_request. This makes typing tighter around the code base as well.
  • IPCProvider did not have request caching turned on. This PR turns that on.

Todo:

  • Add or update documentation related to these changes. This documentation should detail the underlying mechanism of how (and why) to check certain endpoints against a validation threshold so that users know:

    1. When to reach for caching in the first place
    2. If they reach for caching, clarify internal calls can be made to get the necessary validation data
    3. Know how they can tweak the validation threshold to lower the number of internal calls made (or set it to None to cache any and all requests)
  • Add entry to the release notes

  • Clean up commit history

Cute Animal Picture

     __      __
    /  \____/  \
   (    o  o    )
    \    ∇     /
     \  \___/  /
      \_______/
        |   |

@fselmo fselmo force-pushed the non-pos-request-caching branch 2 times, most recently from 9dcbf30 to b5b26de Compare October 11, 2024 16:40
@fselmo fselmo marked this pull request as ready for review October 11, 2024 16:40
@fselmo fselmo force-pushed the non-pos-request-caching branch 2 times, most recently from 9fcd38f to 84d3234 Compare October 11, 2024 16:50
@fselmo fselmo self-assigned this Oct 14, 2024
docs/internals.rst Outdated Show resolved Hide resolved
fselmo added a commit to fselmo/web3.py that referenced this pull request Oct 14, 2024
Copy link
Collaborator

@kclowes kclowes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly left lazy questions and small things, but looks good otherwise!

web3/_utils/caching/caching_utils.py Outdated Show resolved Hide resolved
web3/_utils/caching/request_caching_validation.py Outdated Show resolved Hide resolved
web3/_utils/module_testing/utils.py Show resolved Hide resolved
tests/core/caching-utils/test_request_caching.py Outdated Show resolved Hide resolved
tests/core/caching-utils/test_request_caching.py Outdated Show resolved Hide resolved
- If non-mainnet ethereum, allow the use of an integer value for the
  request cache validation threshold. Pre-configure "safe" default
  values for some common non-mainnet chain ids based on their
  varied finality mechanisms.

- This integer value represents the number of seconds from time.now()
  that the request cache deems as a safe enough time window to allow
  the request to be cached.

- Update the tests to reflect these changes.

Bonus:

- Add a note to the ``request_mocker`` to make sure that mocked results
  are of the correct expected types based on JSON-RPC spec (e.g. hex
  strings instead of ints for numbers, etc.)
- In order to prevent recursion, we need to turn off caching while we
  make internal calls to get the block information we need. Turn it back
  on via a ``finally`` so we are sure to turn it back on.

- Add some comprehensive tests for the now complex request caching
  functionality due to multi-chain support and internal-based
  caching threshold configuration.
fselmo added a commit to fselmo/web3.py that referenced this pull request Oct 14, 2024
- Fix typo in docs.
- Fix test check, be more precise and check cache == 1.
- Remove validation for threshold with chain_id == 1 since
  more chains than mainnet Ethereum can make use of ``finalized``
  and ``safe`` thresholds.
@fselmo fselmo force-pushed the non-pos-request-caching branch 3 times, most recently from 9852a39 to 688ae86 Compare October 15, 2024 15:54
- Fix typo in docs.
- Fix test check, be more precise and check cache == 1.
- Remove validation for threshold with chain_id == 1 since
  more chains than mainnet Ethereum can make use of ``finalized``
  and ``safe`` thresholds.
- Store and reset the current value of ``cache_allowed_requests`` rather than assuming ``True`` / ``False``
Copy link
Contributor

@pacrob pacrob left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just one q, otherwise lgtm!

- Base providers don't implement ``make_request`` and so can't cache
  requests. Remove the request caching decorator from the base provider
  classes.

- Fix typing in the request caching utils, which should now be tighter,
  expecting an ``RPCEndpoint`` instead of a generic ``str``.
@fselmo fselmo merged commit 11d0842 into ethereum:main Oct 15, 2024
71 checks passed
fselmo added a commit that referenced this pull request Oct 15, 2024
- Fix typo in docs.
- Fix test check, be more precise and check cache == 1.
- Remove validation for threshold with chain_id == 1 since
  more chains than mainnet Ethereum can make use of ``finalized``
  and ``safe`` thresholds.
- Store and reset the current value of ``cache_allowed_requests`` rather than assuming ``True`` / ``False``
@fselmo fselmo deleted the non-pos-request-caching branch October 15, 2024 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

get_transaction failed when cache_allowed_requests is set to True.
3 participants