fix: RabbitMQ 4.3.0 compatibility#2531
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2531 +/- ##
=======================================
Coverage 82.49% 82.50%
=======================================
Files 79 79
Lines 10190 10195 +5
Branches 1170 1170
=======================================
+ Hits 8406 8411 +5
Misses 1582 1582
Partials 202 202 ☔ View full report in Codecov by Sentry. |
|
CI pass, nice |
There was a problem hiding this comment.
Pull request overview
Updates Kombu’s pidbox and SimpleBuffer queue declarations to avoid RabbitMQ 4.3.0’s default rejection of transient non-exclusive queues, and adjusts tests/docs accordingly.
Changes:
- Default pidbox (
Mailbox) queues toexclusive=Trueand convert certain channel-level declare failures into anInconsistencyError. - Default
SimpleBufferqueues toexclusive=Trueto preserve ephemeral semantics while remaining compatible with RabbitMQ 4.3.0. - Update unit tests and pidbox reference docs to reflect the new defaults/behavior.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
kombu/pidbox.py |
Default pidbox queues to exclusive; add channel-error handling that raises InconsistencyError. |
kombu/simple.py |
Make SimpleBuffer queues exclusive by default. |
t/unit/test_pidbox.py |
Add/adjust tests for exclusive-default pidbox queues and channel-error behavior. |
t/unit/test_simple.py |
Update tests to assert SimpleBuffer default exclusivity and override behavior. |
docs/reference/kombu.pidbox.rst |
Document queue_durable / queue_exclusive options and the new default. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| conn = self.mailbox.connection | ||
| channel_errors = conn.channel_errors if conn else () | ||
| try: | ||
| return Consumer( | ||
| channel or self.channel, [queue], no_ack=no_ack, | ||
| accept=self.mailbox.accept if accept is None else accept, | ||
| **options | ||
| ) | ||
| except channel_errors as exc: | ||
| raise InconsistencyError( | ||
| W_PIDBOX_IN_USE.format(node=self) | ||
| ) from exc |
There was a problem hiding this comment.
The except channel_errors block converts any channel-level broker failure into InconsistencyError with the “node already using” guidance. channel_errors typically includes cases like ACCESS_REFUSED, PRECONDITION_FAILED, etc., so this can produce a misleading error message and hide the underlying cause. Consider only translating known “exclusive queue already in use” failures (e.g., RESOURCE_LOCKED / specific reply code or exception type), and re-raise other channel errors (or at least include the original exception details in the raised error).
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Summary
RabbitMQ 4.3.0 disables transient non-exclusive queues by default. Kombu's pidbox queues and
SimpleBufferqueues are declared withdurable=False, exclusive=False, which RabbitMQ 4.3.0 rejects with(541) INTERNAL_ERROR.RabbitMQ References
transient_nonexcl_queues— "Covers queues that are both non-durable and non-exclusive, this combination should be avoided. Use durable queues or non-durable exclusive queues."Fix
kombu/pidbox.py— ChangeMailboxdefault fromqueue_exclusive=Falsetoqueue_exclusive=True. Pidbox queues are per-connection, transient control channels — only the declaring connection consumes from them. Messages are published as transient (the exchange setsdelivery_mode='transient'in_get_exchange()), so a durable queue surviving a restart would just leave an empty queue that never auto-deletes, accumulating zombie queues in containerized deployments.exclusive=Truematches the intended lifecycle and is RabbitMQ's recommended alternative for the deprecated pattern.Additionally,
Node.Consumernow catches channel errors (e.g.,RESOURCE_LOCKED) and raisesInconsistencyErrorwith theW_PIDBOX_IN_USEguidance message. With exclusive queues, the existingverify_exclusive/on_declaredsoft-check can't fire (the broker rejects the declare before the callback runs), so this hardens the duplicate-node detection into a proper error.kombu/simple.py— Add'exclusive': TruetoSimpleBuffer.queue_opts.SimpleBufferis documented as "Simple API for ephemeral queues" withdurable=False, auto_delete=True—exclusive=Truematches this ephemeral semantics exactly.Before → After
Mailbox.queue_exclusivedefaultFalseTrueSimpleBuffer.queue_opts{'durable': False, 'auto_delete': True}{'durable': False, 'auto_delete': True, 'exclusive': True}W_PIDBOX_IN_USEwarning viaon_declaredcallbackInconsistencyErrorwithW_PIDBOX_IN_USEmessageWhy
exclusive=True(notdurable=True)delivery_mode='transient') — a durable queue surviving a restart just leaves an empty queuedurable=Truedisablesauto_delete(auto_delete = not self.queue_durable), causing permanent zombie queues in k8s/ECS where each pod gets a unique nodenameexclusive=Trueimpliesauto_delete=Trueper AMQP spec (enforced inentity.py:594-596)ValueErrorguard atpidbox.py:202-206prevents the invaliddurable=True + exclusive=TruecombinationCompanion PR: celery/celery#10290 | Closes #2237