Celery连接rabbitmq出现billiard.exceptions.RestartFreqExceeded: 5 in 1s，解决方案

[2023-08-04 10:28:39,866: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@NQ31', 'copyright': 'Copyright (c) 2007-2022 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 2.0. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 26.0.2', 'product': 'RabbitMQ', 'version': '3.10.8'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2023-08-04 10:28:39,867: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2023-08-04 10:28:39,867: DEBUG/MainProcess] ^-- substep ok
[2023-08-04 10:28:39,867: DEBUG/MainProcess] | Consumer: Starting Events
[2023-08-04 10:28:39,867: DEBUG/MainProcess] Closed channel #1
[2023-08-04 10:28:39,867: WARNING/MainProcess] /home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:498: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

[2023-08-04 10:28:39,868: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@NQ31', 'copyright': 'Copyright (c) 2007-2022 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 2.0. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 26.0.2', 'product': 'RabbitMQ', 'version': '3.10.8'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2023-08-04 10:28:39,869: DEBUG/MainProcess] using channel_id: 1
[2023-08-04 10:28:39,870: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@NQ31', 'copyright': 'Copyright (c) 2007-2022 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 2.0. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 26.0.2', 'product': 'RabbitMQ', 'version': '3.10.8'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2023-08-04 10:28:39,870: DEBUG/MainProcess] using channel_id: 1
[2023-08-04 10:28:39,871: DEBUG/MainProcess] ^-- substep ok
[2023-08-04 10:28:39,871: DEBUG/MainProcess] | Consumer: Starting Heart
[2023-08-04 10:28:39,872: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@NQ31', 'copyright': 'Copyright (c) 2007-2022 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 2.0. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 26.0.2', 'product': 'RabbitMQ', 'version': '3.10.8'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2023-08-04 10:28:39,872: DEBUG/MainProcess] using channel_id: 1
[2023-08-04 10:28:39,873: DEBUG/MainProcess] Closed channel #1
[2023-08-04 10:28:39,874: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@NQ31', 'copyright': 'Copyright (c) 2007-2022 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 2.0. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 26.0.2', 'product': 'RabbitMQ', 'version': '3.10.8'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2023-08-04 10:28:39,874: DEBUG/MainProcess] using channel_id: 1
[2023-08-04 10:28:39,875: DEBUG/MainProcess] ^-- substep ok
[2023-08-04 10:28:39,875: DEBUG/MainProcess] | Consumer: Starting Mingle
[2023-08-04 10:28:39,875: INFO/MainProcess] mingle: searching for neighbors
[2023-08-04 10:28:39,875: DEBUG/MainProcess] using channel_id: 1
[2023-08-04 10:28:39,875: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/amqp/connection.py", line 514, in channel
    return self.channels[channel_id]
KeyError: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 336, in start
    blueprint.start(self)
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/celery/worker/consumer/mingle.py", line 37, in start
    self.sync(c)
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/celery/worker/consumer/mingle.py", line 41, in sync
    replies = self.send_hello(c)
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/celery/worker/consumer/mingle.py", line 54, in send_hello
    replies = inspect.hello(c.hostname, our_revoked._data) or {}
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/celery/app/control.py", line 389, in hello
    return self._request('hello', from_node=from_node, revoked=revoked)
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/celery/app/control.py", line 106, in _request
    return self._prepare(self.app.control.broadcast(
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/celery/app/control.py", line 776, in broadcast
    return self.mailbox(conn)._broadcast(
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/kombu/pidbox.py", line 330, in _broadcast
    chan = channel or self.connection.default_channel
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/kombu/connection.py", line 963, in default_channel
    self._default_channel = self.channel()
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/kombu/connection.py", line 303, in channel
    chan = self.transport.create_channel(self.connection)
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/kombu/transport/pyamqp.py", line 168, in create_channel
    return connection.channel()
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/amqp/connection.py", line 517, in channel
    channel.open()
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/amqp/connection.py", line 525, in drain_events
    while not self.blocking_read(timeout):
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/amqp/connection.py", line 530, in blocking_read
    frame = self.transport.read_frame()
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/amqp/transport.py", line 294, in read_frame
    frame_header = read(7, True)
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/amqp/transport.py", line 627, in _read
    s = recv(n - len(rbuf))
ConnectionResetError: [Errno 104] Connection reset by peer
[2023-08-04 10:28:39,876: DEBUG/MainProcess] Closed channel #1
[2023-08-04 10:28:39,876: WARNING/MainProcess] /home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:386: CPendingDeprecationWarning: 
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered
back to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss
setting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.

  warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)

[2023-08-04 10:28:39,876: DEBUG/MainProcess] | Consumer: Restarting event loop...
[2023-08-04 10:28:39,876: DEBUG/MainProcess] | Consumer: Restarting Gossip...
[2023-08-04 10:28:39,876: DEBUG/MainProcess] | Consumer: Restarting Control...
[2023-08-04 10:28:39,876: DEBUG/MainProcess] | Consumer: Restarting Tasks...
[2023-08-04 10:28:39,876: DEBUG/MainProcess] | Consumer: Restarting Mingle...
[2023-08-04 10:28:39,876: DEBUG/MainProcess] | Consumer: Restarting Heart...
[2023-08-04 10:28:39,877: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@NQ31', 'copyright': 'Copyright (c) 2007-2022 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 2.0. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 26.0.2', 'product': 'RabbitMQ', 'version': '3.10.8'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2023-08-04 10:28:39,878: DEBUG/MainProcess] using channel_id: 1
[2023-08-04 10:28:39,878: DEBUG/MainProcess] | Consumer: Restarting Events...
[2023-08-04 10:28:39,878: DEBUG/MainProcess] | Consumer: Restarting Connection...
[2023-08-04 10:28:39,878: CRITICAL/MainProcess] Frequent restarts detected: RestartFreqExceeded('5 in 1s')
Traceback (most recent call last):
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 326, in start
    self._restart_state.step()
  File "/home/nq/.virtualenvs/env_django/lib/python3.9/site-packages/billiard/common.py", line 153, in step
    raise self.RestartFreqExceeded("%r in %rs" % (R, self.maxT))
billiard.exceptions.RestartFreqExceeded: 5 in 1s

一开始用的rabbitmq是3.10版本，尝试了网上各种方法，一直没有解决这个问题，后面
根据rabbitmq官网重新安装最新版本3.12就可以解决这个问题了