docs: synchronize foundational and user documentation with failover stack

Updated README, ARCHITECTURE, and CONFIGURATION to reflect the completion of the
automatic failover, quorum durability, and cluster membership work.

- Unified failover documentation by merging controlled and automatic paths into website/docs/concepts/failover.mdx.
- Added ChaosFailoverProof mission to transit-cli to provide a live verification path for ElectionMonitor.
- Integrated chaos-failover-proof into 'just screen' and the default verification suite.
- Fixed a bug in LocalEngine::is_leader that incorrectly defaulted to true when a consensus provider was configured but no handle was bound.
- Updated AGENTS.md and GUIDE.md status to match the latest implementation milestones.

• [MSN] No active missions on the board
• [EXC] Board idle, no stories queued or active
• [HLT] 2 warnings, no structural errors detected

Author: rupurt

AGENTS.md

@@ -51,18 +51,18 @@ Follow the formal procedural loops and checklists defined in:
 
 Keep `just screen` as the default human proof path. If verification gets richer, improve that path instead of making the operator memorize an expanding command list.
 
-## Current Status (2026-03-25)
+## Current Status (2026-03-30)
 
 - **Kernel Done:** Single-node local engine with branch, merge, and tiered storage verified.
 - **Server Done:** Networked daemon with framed protocol, remote CLI, and tail sessions verified.
 - **Integrity Done:** Verifiable lineage primitives, manifest roots, and checkpoints landed.
 - **Materialization Done:** Branch-aware materialization kernel and Prolly Tree snapshots landed.
-- **Consensus Slice Done:** Initial consensus kernel and leader-enforcement slice landed.
-- **Proof Ready:** `just screen` covers local, tiered, networked, integrity, and materialization end-to-end flows.
+- **Consensus Done:** Lease-backed consensus, controlled failover, and leader enforcement verified.
+- **Quorum & Failover Done:** Quorum durability mode, cluster membership, automatic leader election, and election monitoring verified. A follower can automatically acquire an expired lease and become the writable primary; the former primary is fenced.
+- **Proof Ready:** `just screen` covers local, tiered, networked, integrity, materialization, and controlled failover end-to-end flows.
 
 ## Next Missions
 
-- **Replication Planning:** Decomposing the staged multi-node replication model into voyages and ready stories.
 - **Board Hygiene:** Keeping mission/epic intent, generated artifacts, and pacemaker state aligned with execution.
 - **Client Libs:** Promoting external usage via a dedicated Rust client first; other language bindings can follow later.
 

ARCHITECTURE.md

@@ -125,7 +125,7 @@ Server mode exposes the same storage engine behind a network API:
 
 The server should not invent a second storage format or branch model.
 
-The first implementation step is a thin daemon bootstrap that opens the shared engine and binds a listener. The current server slice now layers provisional remote root creation, append, read, snapshot-tail, branch creation, merge creation, and lineage inspection operations on top of that bootstrap, wrapped in a framed request/response envelope with correlation IDs plus explicit acknowledgement and error semantics. Tail streaming now uses logical session IDs with `open/poll/cancel` operations and credit-based delivery so the semantics do not collapse into one socket or underlay assumption. The first CLI client surface now mirrors those remote workflows directly, while richer client surfaces remain downstream. The public server surface remains explicitly single-node. Separately, the shared engine now has a bounded controlled failover slice for published-frontier readiness, explicit lease handoff, and former-primary fencing, but that slice still sits below quorum acknowledgement, automatic election, and multi-primary behavior.
+The first implementation step is a thin daemon bootstrap that opens the shared engine and binds a listener. The current server slice now layers provisional remote root creation, append, read, snapshot-tail, branch creation, merge creation, and lineage inspection operations on top of that bootstrap, wrapped in a framed request/response envelope with correlation IDs plus explicit acknowledgement and error semantics. Tail streaming now uses logical session IDs with `open/poll/cancel` operations and credit-based delivery so the semantics do not collapse into one socket or underlay assumption. The first CLI client surface now mirrors those remote workflows directly, while richer client surfaces remain downstream. The shared engine now has a full failover stack: controlled handoff, automatic leader election via `ElectionMonitor`, quorum-based durability, and cluster membership. Multi-primary behavior remains explicitly out of scope.
 
 The transport boundary is also explicit: `transit` defines an application protocol above the transport layer. TCP, QUIC, or other ordinary transports can carry that protocol, and secure meshes such as WireGuard remain optional deployment underlays rather than protocol replacements.
 
@@ -182,7 +182,8 @@ Suggested durability modes:
 
 - `memory`: acknowledged after in-memory acceptance, for tests only
 - `local`: acknowledged after local durable write
-- `replicated`: acknowledged after the published handoff frontier is durable enough for read-only replica catch-up and promotion readiness; it does not imply follower hydration, quorum acknowledgement, or automatic failover
+- `replicated`: acknowledged after the published handoff frontier is durable enough for read-only replica catch-up and promotion readiness
+- `quorum`: acknowledged after a majority of configured cluster peers have confirmed receipt
 - `tiered`: acknowledged only after the relevant segment state is durable in the remote tier
 
 ## Read Path
@@ -228,9 +229,18 @@ The initial reference model should be simple:
 - acknowledged records are immutable
 - recovery must never expose unacknowledged bytes as committed history
 
-The first controlled failover slice preserves that model by allowing a caught-up read-only replica to become the writable primary only through explicit lease handoff. Former primaries are fenced after handoff so stale leaders cannot continue acknowledged writes. This slice remains explicitly below quorum acknowledgement, election, and multi-primary behavior.
+The failover model preserves that invariant through three complementary components:
 
-Any move toward replicated or multi-writer semantics must preserve those invariants and define conflict rules directly.
+- **ClusterMembership:** Nodes discover each other and maintain heartbeats to calculate quorum size.
+- **ElectionMonitor:** A background worker that polls for lease expiration and triggers automatic leader election via the `ConsensusProvider`.
+- **ObjectStoreConsensus:** A provider that uses optimistic locking on the remote tier to ensure that only one node can acquire a writable lease at a time.
+
+Failover is supported through two paths:
+
+- **Controlled failover:** A caught-up read-only replica becomes the writable primary through explicit lease handoff. The former primary is fenced after handoff.
+- **Automatic failover:** The `ElectionMonitor` detects primary failure (lease expiry) and triggers automatic acquisition by an eligible follower.
+
+Both paths preserve the one-writer-per-stream-head invariant. Multi-primary behavior remains explicitly out of scope.
 
 ## Processing And Materialization
 
@@ -326,7 +336,8 @@ That means lineage metadata should be cheap to create and easy to query.
 
 These areas are important but should stay explicit future work until designed:
 
-- distributed consensus and cross-node replication
+- multi-primary or multi-writer semantics
+- dynamic cluster rebalancing and automatic data sharding
 - compaction or projection layers above immutable history
 - authn/authz and multi-tenant isolation
 - query surfaces beyond ordered log replay and tailing

CONFIGURATION.md

@@ -148,15 +148,18 @@ The current bootstrap implementation wires `data_dir` through `transit server ru
 
 ### `[replication]`
 
-Replication is deferred scope, but the config surface should be explicit once introduced.
+Replication and failover settings for clustered deployments.
 
 | Key | Type | Default | Description |
 |-----|------|---------|-------------|
-| `mode` | String | `"single-node"` | Initial deployment model. |
-| `sync_quorum` | Integer | `1` | Number of nodes required for future replicated ack. |
-| `peer_urls` | Array | `[]` | Planned peer list for future replicated topologies. |
-
-Until replication exists, `single-node` should remain the only supported value.
+| `mode` | String | `"single-node"` | Deployment model: `single-node` or `cluster`. |
+| `node_id` | String | null | Unique identity for this node (overrides `[node].id`). |
+| `consensus_root` | String | null | Object-store path used for shared leases and elections. |
+| `lease_duration_secs` | Integer | `10` | TTL for the primary lease. |
+| `election_poll_interval_ms` | Integer | `1000` | How often the `ElectionMonitor` checks lease health. |
+| `quorum_size` | Integer | `1` | Number of nodes required for `quorum` durability. |
+
+`durability` mode `quorum` depends on these settings to discover peers and calculate the required majority.
 
 ### `[telemetry]`
 

GUIDE.md

@@ -44,7 +44,7 @@ Examples:
 
 At the current bootstrap stage, the shared-engine server exposes provisional remote root creation, append, read, snapshot-tail, branch creation, merge creation, and lineage inspection operations through `transit-core::server::RemoteClient`. The first wire shape now includes request correlation plus explicit acknowledgement and error envelopes, and the first tail-session model now uses logical `open/poll/cancel` operations with credit-based delivery rather than assuming one long-lived socket. The `transit server` CLI namespace now mirrors those workflows directly with `create-root`, `append`, `read`, `branch`, `merge`, `lineage`, `tail-open`, `tail-poll`, and `tail-cancel`. The surface is still explicitly single-node; replication-aware behavior is downstream work, and secure transports such as WireGuard remain optional underlays instead of becoming the `transit` protocol.
 
-The first replicated failover slice is proof-oriented and intentionally bounded. A caught-up read-only replica can be promoted through an explicit lease handoff, the former primary is fenced, and the proof surface makes the non-claims explicit: this is not quorum acknowledgement, automatic election, or multi-primary behavior.
+The failover stack now supports two paths. **Controlled failover** lets an operator explicitly hand off the writable lease from a primary to a caught-up read-only replica. **Automatic failover** uses an `ElectionMonitor` that detects an expired primary lease and triggers a follower to acquire it via optimistic locking; only one node wins the race, and the former primary is fenced. The engine also supports a `quorum` durability mode where appends block until a majority of configured cluster peers have acknowledged, and a `ClusterMembership` surface for node discovery and quorum calculation. Multi-primary behavior remains explicitly out of scope.
 
 ## Modeling Conversations
 

Justfile

@@ -28,7 +28,7 @@ screen:
         cd "$repo_root"
 
         rm -rf "$screen_root"
-        mkdir -p "$screen_root/object-store" "$screen_root/local-engine" "$screen_root/integrity" "$screen_root/materialization" "$screen_root/tiered-engine" "$screen_root/controlled-failover" "$screen_root/networked-server"
+        mkdir -p "$screen_root/object-store" "$screen_root/local-engine" "$screen_root/integrity" "$screen_root/materialization" "$screen_root/tiered-engine" "$screen_root/controlled-failover" "$screen_root/chaos-failover" "$screen_root/networked-server"
 
         announce "Build workspace"
         just build
@@ -38,6 +38,8 @@ screen:
         just run mission tiered-engine-proof --root "$screen_root/tiered-engine"
         announce "Prove controlled failover"
         just run mission controlled-failover-proof --root "$screen_root/controlled-failover"
+        announce "Prove chaos failover"
+        just run mission chaos-failover-proof --root "$screen_root/chaos-failover"
         announce "Prove networked server"
         just run mission networked-server-proof --root "$screen_root/networked-server"
         announce "Prove integrity proof"

README.md

@@ -93,16 +93,14 @@ This repository is at the bootstrap stage.
 
 Today it contains:
 
-- a Rust workspace with `transit-core` and `transit-cli`
-- a Nix flake and Rust toolchain bootstrap
-- a `Justfile` with a human-facing `just screen` verification path for local-engine proof, tiered publication/restore proof, controlled failover proof, networked single-node server proof, integrity proof, materialization proof, object-store probing, and the current Keel board view
-- a local durable engine that can append, replay, branch, merge, recover from trailing uncommitted active-head bytes, publish rolled immutable segments to object storage, and cold-restore published history from remote manifests
-- an initial shared-engine server bootstrap that can open the same local engine, bind a daemon listener, shut down deterministically, and serve provisional remote root creation, append/read/tail, branch/merge, and lineage-inspection operations through a framed request/response envelope with correlation IDs, explicit acknowledgement and error semantics, and logical tail sessions with credit-based delivery, without introducing a second storage path
-- a first CLI client surface for remote root creation, append, read, branch, merge, lineage inspection, and logical tail-session workflows
-- a first Rust client library surface plus a native proof example that exercises create_root, append, read, tail, branch, merge, and lineage against a locally started server
-- a first controlled failover proof path that shows promotion readiness, explicit lease handoff, and former-primary fencing while keeping local, replicated, tiered, quorum, and multi-primary guarantees explicit
-- a first networked mission proof path that validates the live single-node server and keeps the `transit` protocol explicitly distinct from optional secure underlays such as WireGuard
-- an initial `object_store` integration with a filesystem probe command
+- **Local Engine:** Durable local append, replay, branch, merge, and crash recovery with trailing-byte truncation.
+- **Tiered Storage:** Native publication to object storage and cold restore from remote manifests.
+- **Failover Stack:** Controlled handoff with lease fencing, automatic leader election via `ElectionMonitor`, and quorum-based durability.
+- **Networked Server:** Single-node daemon bootstrap with a framed request/response protocol and logical tail sessions.
+- **Integrity:** Staged verification from checksums to manifest roots and lineage checkpoints.
+- **Materialization:** Incremental processing with Prolly Tree snapshots and checkpoint-based resume.
+- **Clients:** Native Rust client library and a feature-complete CLI for operations and proofs.
+- **Verification:** A unified `just screen` path that runs the full suite of human-verifiable missions.
 
 The implementation work now has a real scaffold to grow from instead of needing to reverse-engineer direction later.