Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Don't assume ordering of ThreadPoolExecutor submissions #5052

Merged
merged 4 commits into from
Mar 13, 2024

Conversation

holmanb
Copy link
Member

@holmanb holmanb commented Mar 13, 2024

Proposed Commit Message

fix: Don't assume ordering of ThreadPoolExecutor submissions

This test breaks in Python 3.13 and was never guaranteed to
succeed in earlier versions of Python but apparently did due
to the way that cPython was implementated previously.

Additional Context

We've had some flaky test failures for a bit now: [example]

This test was never guaranteed to succeed, it just apparently did because of cPython ThreadpoolExecutor implementation. Python 3.13 brings lots of threading changes - it is unsurprising that this test now fails occasionally.

self = <tests.unittests.test_url_helper.TestDualStack object at 0x7f07de5e12e0>

    def test_dual_stack_staggered(self):
        """Assert expected call intervals occur"""
        stagger = 0.1
        with mock.patch(M_PATH + "_run_func_with_delay") as delay_func:
            dual_stack(
                lambda x, _y: x,
                ["you", "and", "me", "and", "dog"],
                stagger_delay=stagger,
                timeout=1,
            )
    
            # ensure that stagger delay for each subsequent call is:
            # [ 0 * N, 1 * N, 2 * N, 3 * N, 4 * N, 5 * N] where N = stagger
            # it appears that without an explicit wait/join we can't assert
            # number of calls
            for delay, call_item in enumerate(delay_func.call_args_list):
                _, kwargs = call_item
>               assert stagger * delay == kwargs.get("delay")
E               AssertionError: assert (0.1 * 1) == 0.2
E                +  where 0.2 = <built-in method get of dict object at 0x7f077cee4d40>('delay')
E                +    where <built-in method get of dict object at 0x7f077cee4d40> = {'addr': 'me', 'delay': 0.2, 'event': <threading.Event at 0x7f07ac855700: set>, 'func': <function TestDualStack.test_dual_stack_staggered.<locals>.<lambda> at 0x7f077cef02c0>, ...}.get

_          = ()
call_item  = call(func=<function TestDualStack.test_dual_stack_staggered.<locals>.<lambda> at 0x7f077cef02c0>, addr='me', timeout=1, event=<threading.Event at 0x7f07ac855700: set>, delay=0.2)
delay      = 1
delay_func = <MagicMock name='_run_func_with_delay' id='139670934497200'>
kwargs     = {'addr': 'me',
 'delay': 0.2,
 'event': <threading.Event at 0x7f07ac855700: set>,
 'func': <function TestDualStack.test_dual_stack_staggered.<locals>.<lambda> at 0x7f077cef02c0>,
 'timeout': 1}
self       = <tests.unittests.test_url_helper.TestDualStack object at 0x7f07de5e12e0>
stagger    = 0.1

Test Steps

It passes for me locally and I don't have a Python 3.13 interpreter locally so... did ci pass?

Copy link
Member

@TheRealFalcon TheRealFalcon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! If I run this test in a loop on main I get a failure within 10 tries. I have been running this branch in a loop while reviewing the changes and we're up to 200 runs without failure.

It passes for me locally and I don't have a Python 3.13 interpreter locally so... did ci pass?

pyenv is your friend 🙂

@TheRealFalcon TheRealFalcon merged commit f7c1c76 into canonical:main Mar 13, 2024
29 checks passed
TheRealFalcon pushed a commit to TheRealFalcon/cloud-init that referenced this pull request Mar 20, 2024
…cal#5052)

This test breaks in Python 3.13 and was never guaranteed to
succeed in earlier versions of Python but apparently did due
to the way that cPython was implemented previously.
@TheRealFalcon TheRealFalcon mentioned this pull request Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants