CHANNEL_ERROR KeyError: None

[radiantone]

Hi,
Out of the blue, I am getting the below exception. I am using rabbitmq 3.9-management docker, redis 6.2.6 docker and celery 5.2.1

I attached a simple test program that reproduces the problem. Worked fine yesterday. Today, broken.

[2021-12-19 15:21:49,021: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/amqp/connection.py", line 510, in channel
    return self.channels[channel_id]
KeyError: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 326, in start
    blueprint.start(self)
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/celery/worker/pidbox.py", line 52, in start
    self.node.channel = c.connection.channel()
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/kombu/connection.py", line 282, in channel
    chan = self.transport.create_channel(self.connection)
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/kombu/transport/pyamqp.py", line 166, in create_channel
    return connection.channel()
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/amqp/connection.py", line 513, in channel
    channel.open()
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/amqp/channel.py", line 436, in open
    return self.send_method(
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/amqp/abstract_channel.py", line 66, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/amqp/abstract_channel.py", line 86, in wait
    self.connection.drain_events(timeout=timeout)
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/amqp/connection.py", line 521, in drain_events
    while not self.blocking_read(timeout):
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/amqp/connection.py", line 527, in blocking_read
    return self.on_inbound_frame(frame)
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/amqp/connection.py", line 533, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/amqp/abstract_channel.py", line 143, in dispatch_method
    listener(*args)
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/amqp/connection.py", line 663, in _on_close
    raise error_for_code(reply_code, reply_text,
amqp.exceptions.ChannelNotOpen: Channel.open: (504) CHANNEL_ERROR - second 'channel.open' seen
[2021-12-19 15:21:49,021: WARNING/MainProcess] /home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:361: CPendingDeprecationWarning: 
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered back to the queue.
You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss setting.
In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.

  warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)

[2021-12-19 15:21:49,022: CRITICAL/MainProcess] Frequent restarts detected: RestartFreqExceeded('5 in 1s')
Traceback (most recent call last):
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 320, in start
    self._restart_state.step()
  File "/home/darren/PycharmProjects/pyfi/venv4/lib/python3.9/site-packages/billiard/common.py", line 165, in step
    raise self.RestartFreqExceeded("%r in %rs" % (R, self.maxT))
billiard.exceptions.RestartFreqExceeded: 5 in 1s```

[open-collective-bot]

Hey @radiantone 👋,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

[radiantone]

Here is the test program and packages

from celery import Celery

backend='redis://localhost'
broker='pyamqp://localhost'

celery = Celery('pyfi', backend=backend, broker=broker)

worker = celery.Worker(hostname='test@phoenix',
                        backend=backend,
                        broker=broker,
                        beat=False,
                        uid='pyfi',
                        without_mingle=True,
                        without_gossip=True,
                        concurrency=int(1)
                    )

worker.start()

amqp==5.0.8
billiard==3.6.4.0
celery==5.2.1
click==8.0.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
Deprecated==1.2.13
kombu==5.2.2
prompt-toolkit==3.0.24
pytz==2021.3
redis==4.0.2
six==1.16.0
vine==5.0.0
wcwidth==0.2.5
wrapt==1.13.3

[radiantone]

Source issue and fix is here celery/py-amqp#382