-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix executor shutdown #392
base: master
Are you sure you want to change the base?
Conversation
cd0361c
to
ad9ffaa
Compare
Avoid exception: Traceback (most recent call last): File "lib/python3.13/threading.py", line 1041, in _bootstrap_inner self.run() ~~~~~~~~^^ File "cassandra/cluster.py", line 4429, in run future = self._executor.submit(fn, *args, **kwargs) File "lib/python3.13/concurrent/futures/thread.py", line 171, in submit Error: raise RuntimeError('cannot schedule new futures after shutdown') RuntimeError: cannot schedule new futures after shutdown
ad9ffaa
to
63110b0
Compare
What is the task that is being scheduled? I know about this error, but I didn't fix it that way before because it is a wrong way in my opinion. We should first end all tasks / threads (or whatever are they called here), and only after they are stopped the scheduler can be stopped. Ignoring this error is not fixing the issue but just masking it. I envision it could hurt us in the future - perhaps the queries that were not finished will be forever waiting? Or there will be something that schedules a task, and awaits its result, causing a hang? I'd like to avoid issues similar to cassandra-stress being stuck on shutdown. |
It is actually already done, here python-driver/cassandra/cluster.py Lines 4399 to 4407 in 63110b0
And here it is python-driver/cassandra/cluster.py Lines 1888 to 1895 in 63110b0
Which means that given issue can happen only when shutdown procedure does not go as it should. |
Wait, so the error can happen in one of 3 conditions:
The third case is not important with regard to warnings, I think it is ok to print them. The first case is why the warning should not be ignored imo. If we fix second case, and don't kill the process, and still see the warning, then it indicates a bug which we should fix, right? |
Correct, Let's do this:
WDYT ? |
Usually
Cluster.executor
is getting shutdown afterCluster.scheduler
, but in some cases executor is getting shutdown earlier, probably whenCluster.executor
is manipulated directly or when process is getting killed andCluster
instance is still alive.When it happens you will see this exception:
This PR makes it handle this case gracefully not scheduling anything and not failing.
Pre-review checklist
I added relevant tests for new features and bug fixes.I have adjusted the documentation in./docs/source/
.I added appropriateFixes:
annotations to PR description.