Ensure consistency in retry logic

[Gusarich]

We need to make sure failures are handled properly and consistently across all stages

[oxgeneral]

We tackled the same problem in ORCH — a TypeScript CLI orchestrator for Claude Code, Codex, and Cursor. Sharing the approach in case it's useful.

The core insight: retry logic becomes much cleaner when failures are first-class states in a state machine rather than exception paths. In ORCH, tasks flow through:

todo → in_progress → review → done
                  ↘ retrying → in_progress  (auto-retry loop)
                  ↘ failed                  (terminal after max attempts)

The retrying state is what gives you consistency — every failure handler writes the same transition, whether it's a timeout, a crashed process, or a non-zero exit. The orchestrator's reconcile loop checks for stalled in_progress tasks and pushes them to retrying automatically.

Key properties that helped us:

• Idempotent transitions — canTransition(current, next) validates every move, so you can't accidentally skip states
• Retry counter on the task — attempts: number incremented each time, terminal at maxAttempts (configurable per-agent)
• State is the source of truth — even if the process crashes mid-run, the next tick reconciles from state, not from process table

Pitaya's Docker-per-agent model is actually a nice complement to this — each retry gets a clean container. The state machine handles when to retry; Docker handles where.

Happy to share the transitions impl if useful: src/domain/transitions.ts

[Gusarich]

@oxgeneral sounds like a clean approach! I'm not actively working on this project though. Came to a conclusion that setups should be as simple as possible and this whole orchestration idea is overcomplicated.