Skip to content

fix: RabbitMQ 4.3.0 compatibility#2531

Open
Nusnus wants to merge 3 commits intocelery:mainfrom
Nusnus:rabbitmq-430-fix-2
Open

fix: RabbitMQ 4.3.0 compatibility#2531
Nusnus wants to merge 3 commits intocelery:mainfrom
Nusnus:rabbitmq-430-fix-2

Conversation

@Nusnus
Copy link
Copy Markdown
Member

@Nusnus Nusnus commented Apr 27, 2026

Summary

RabbitMQ 4.3.0 disables transient non-exclusive queues by default. Kombu's pidbox queues and SimpleBuffer queues are declared with durable=False, exclusive=False, which RabbitMQ 4.3.0 rejects with (541) INTERNAL_ERROR.

RabbitMQ References

  • Release Notes (Broadcom TechDocs): "Deprecated Features are Now Disabled by Default — This includes non-durable (transient) non-exclusive queues: attempts to declare a queue with that property combination are rejected by default. Use durable queues, transient exclusive queues, or durable queues with a queue TTL instead."
  • Deprecated Features List: transient_nonexcl_queues"Covers queues that are both non-durable and non-exclusive, this combination should be avoided. Use durable queues or non-durable exclusive queues."
  • Deprecation Announcement (August 2021): Original deprecation notice for RabbitMQ 4.0+

Fix

kombu/pidbox.py — Change Mailbox default from queue_exclusive=False to queue_exclusive=True. Pidbox queues are per-connection, transient control channels — only the declaring connection consumes from them. Messages are published as transient (the exchange sets delivery_mode='transient' in _get_exchange()), so a durable queue surviving a restart would just leave an empty queue that never auto-deletes, accumulating zombie queues in containerized deployments. exclusive=True matches the intended lifecycle and is RabbitMQ's recommended alternative for the deprecated pattern.

Additionally, Node.Consumer now catches channel errors (e.g., RESOURCE_LOCKED) and raises InconsistencyError with the W_PIDBOX_IN_USE guidance message. With exclusive queues, the existing verify_exclusive / on_declared soft-check can't fire (the broker rejects the declare before the callback runs), so this hardens the duplicate-node detection into a proper error.

kombu/simple.py — Add 'exclusive': True to SimpleBuffer.queue_opts. SimpleBuffer is documented as "Simple API for ephemeral queues" with durable=False, auto_delete=Trueexclusive=True matches this ephemeral semantics exactly.

Before → After

Before After
Mailbox.queue_exclusive default False True
SimpleBuffer.queue_opts {'durable': False, 'auto_delete': True} {'durable': False, 'auto_delete': True, 'exclusive': True}
Duplicate node detection W_PIDBOX_IN_USE warning via on_declared callback InconsistencyError with W_PIDBOX_IN_USE message

Why exclusive=True (not durable=True)

  • Pidbox messages are transient (exchange delivery_mode='transient') — a durable queue surviving a restart just leaves an empty queue
  • durable=True disables auto_delete (auto_delete = not self.queue_durable), causing permanent zombie queues in k8s/ECS where each pod gets a unique nodename
  • The pidbox lifecycle is a natural fit for exclusive queues: one connection declares it, only that connection consumes, the queue dies with its consumer
  • exclusive=True implies auto_delete=True per AMQP spec (enforced in entity.py:594-596)
  • The ValueError guard at pidbox.py:202-206 prevents the invalid durable=True + exclusive=True combination

Companion PR: celery/celery#10290 | Closes #2237

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 27, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.50%. Comparing base (339fb58) to head (08eacaa).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2531   +/-   ##
=======================================
  Coverage   82.49%   82.50%           
=======================================
  Files          79       79           
  Lines       10190    10195    +5     
  Branches     1170     1170           
=======================================
+ Hits         8406     8411    +5     
  Misses       1582     1582           
  Partials      202      202           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Nusnus
Copy link
Copy Markdown
Member Author

Nusnus commented Apr 27, 2026

CI pass, nice

@auvipy auvipy requested a review from Copilot April 28, 2026 04:31
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates Kombu’s pidbox and SimpleBuffer queue declarations to avoid RabbitMQ 4.3.0’s default rejection of transient non-exclusive queues, and adjusts tests/docs accordingly.

Changes:

  • Default pidbox (Mailbox) queues to exclusive=True and convert certain channel-level declare failures into an InconsistencyError.
  • Default SimpleBuffer queues to exclusive=True to preserve ephemeral semantics while remaining compatible with RabbitMQ 4.3.0.
  • Update unit tests and pidbox reference docs to reflect the new defaults/behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
kombu/pidbox.py Default pidbox queues to exclusive; add channel-error handling that raises InconsistencyError.
kombu/simple.py Make SimpleBuffer queues exclusive by default.
t/unit/test_pidbox.py Add/adjust tests for exclusive-default pidbox queues and channel-error behavior.
t/unit/test_simple.py Update tests to assert SimpleBuffer default exclusivity and override behavior.
docs/reference/kombu.pidbox.rst Document queue_durable / queue_exclusive options and the new default.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread kombu/pidbox.py
Comment on lines +75 to +86
conn = self.mailbox.connection
channel_errors = conn.channel_errors if conn else ()
try:
return Consumer(
channel or self.channel, [queue], no_ack=no_ack,
accept=self.mailbox.accept if accept is None else accept,
**options
)
except channel_errors as exc:
raise InconsistencyError(
W_PIDBOX_IN_USE.format(node=self)
) from exc
Copy link

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The except channel_errors block converts any channel-level broker failure into InconsistencyError with the “node already using” guidance. channel_errors typically includes cases like ACCESS_REFUSED, PRECONDITION_FAILED, etc., so this can produce a misleading error message and hide the underlying cause. Consider only translating known “exclusive queue already in use” failures (e.g., RESOURCE_LOCKED / specific reply code or exception type), and re-raise other channel errors (or at least include the original exception details in the raised error).

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Nusnus bro can you cross check this, please?

Comment thread t/unit/test_pidbox.py Outdated
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Change default value for Durable flag on Queue creation for pidbox (RabbitMQ deprecated feature transient_nonexcl_queues)

3 participants