HCD-595: Fix hang due to unflushed partial batches

[wolfiestyle]

HCD-595

Summary

• Fix flush_and_sync_group hanging when recursive async handlers complete after the final flush, leaving partial batches
  (< batch_size) unprocessed in batched receivers
• Replace the single flush + wait_idle with a convergence loop that repeatedly flushes and checks is_idle, with a
  64-iteration safety fuse
• Add a regression test that reproduces the exact race condition: recursive handlers with async delays send messages to a batched receiver with a large batch_size, creating partial batches after the flush

Root Cause

1. flush_and_sync_group performed one final flush then called wait_idle
2. Slow recursive handlers completed after that flush and sent new messages to batched receivers
3. These messages formed partial batches (count < batch_size) that were never auto-triggered
4. The group processing counter stayed > 0 because the batch handler never ran
5. wait_idle blocked forever

Test Plan

• New test test_flush_and_sync_group_late_partial_batch_after_recursive_handlers reproduces the hang with a 5-second timeout assertion
• cargo test passes
• cargo clippy clean

Summary by CodeRabbit

• Bug Fixes

  • Fixed a hang condition that occurred with recursive handlers emitting partial batches. Improved synchronization logic to ensure all queued messages are properly flushed before completing operations.

• Tests

  • Added regression test to prevent recurrence of race condition with late-arriving partial batches.

[coderabbitai]

Walkthrough

The flush_and_sync_group implementation is refactored to use an iterative drain loop that repeatedly flushes partial batches and checks for idle state, replacing the previous single flush-and-wait approach. A corresponding test validates the fix against a race condition involving recursive handlers and partial batches.

Changes

Cohort / File(s)	Summary
Flush and Sync Implementation src/lib.rs	Modified flush_and_sync_group to iterate drain-flush-check cycles (max 64 iterations) with is_idle checks after each drain, allowing in-flight handlers to progress between iterations. Adds convergence warning logging and maintains group counter tracking.
Race Condition Test tests/test_groups.rs	Added comprehensive test test_flush_and_sync_group_late_partial_batch_after_recursive_handlers that reproduces and validates fix for hang scenario caused by partial batches from recursive, delayed handlers emitting to large batched receivers.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Poem

🐰 A loop of drains, a convergence dance,
Recursive hands in iteration's trance,
Partial batches flushed with grace,
No more hangs in this sync'd space!
From single flush to fifty-four,
Progress yields forevermore. 🔄

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'HCD-595: Fix hang due to unflushed partial batches' clearly and specifically describes the main fix: addressing a hang caused by unflushed partial batches in flush_and_sync_group.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch alanhernandez/hcd-595-fix-hang-on-scrape_urls

---

No actionable comments were generated in the recent review. 🎉

Tip

Issue Planner is now in beta. Read the docs and try it out! Share your feedback on Discord.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[claude]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.