LedgerDB: implement predictable snapshotting by amesgen · Pull Request #1575 · IntersectMBO/ouroboros-consensus

amesgen · 2025-06-30T08:31:59Z

Based on top of #1513, see there for the relation to this change.

Note that we need #1573 before this can be released in a node.

This PR is intended to be reviewed commit-by-commit.

This PR replaces the previous logic for when to create snapshots (It would be possible to preserve it, but I don't see a big motivation). Concretely, SnapshotFrequencyArgs contains

sfaInterval :: SlotNo: Create snapshots every sfaInterval many slots.

Default: 2*k = 4320 slots, so 72 min on mainnet as before.
sfaOffset :: SlotNo: Allows to determine the offset of where snapshots are taken, see below.

Default: 0
sfaRateLimit :: DiffTime: A minimum duration between snapshots (used to avoid excessive snapshots while syncing).

Default: 10 minutes (previous value was 6 minutes, which seemed a bit low, so I increased it somewhat. Maybe it should be increased even more now that we no longer have the substantialAmountOfBlocksWereProcessed check.)

Concretely, the node will try to create snapshots for the last immutable blocks before the slots

sfaOffset, sfaOffset + sfaInterval, sfaOffset + 2 * sfaInterval, sfaOffset + 3 * sfaInterval, ...

but can skip creating some of these depending on the sfaRateLimit (which can be disabled by setting it to a non-positive value). Also see the Haddocks.

For example, setting sfaInterval = 10*2160*20 (one mainnet Shelley epoch) and sfaOffset = 172800 will cause the node to create snapshots for the last block in every Shelley epoch (because the first Shelley slot is 4492800, and 4492800 `mod` (10*2160*20) = 172800. By tweaking sfaOffset, one can take snapshots eg right before the midway point in each epoch.

There is some code that could be shared between V1 and V2 (already even before this PR), but given the upcoming removal of V1, this seems acceptable for now.

Also includes a test running a ChainDB in IOSim to test that everything is hooked up correctly (in particular regarding the background threads).

See #1513 (comment) for sync regression tests.

amesgen · 2025-06-30T08:38:05Z

I integrated this into 10.5.1 for testing:

Consensus branch: amesgen/predictable-snapshots-10.5
Node branch: amesgen/test-snapshot-stuff

In the Node config, the following config would result in the node creating a snapshot for the last block in every Shelly epoch (having 10*2160/20 = 432000 slots):

LedgerDB:
  Backend: V2InMemory # or V1LMDB
  # Number of slots in a Shelley epoch, so we will create snapshots
  # precisely for the last block in each Shelley epoch
  SnapshotInterval: 432000
  # Due to Byron, the first slots of Shelley epochs are offset by this amount
  SlotOffset: 172800
  # Disable the rate limit, to be sure that we definitely create snapshots
  # for all epochs
  RateLimit: 0

Currently, this treats SnapshotInterval (which previously was a number of seconds) as a number of slots. This is convenient, and should be fine for mainnet AFAICT, but maybe we still want to add some kind of warning (see also the TODO in the code there).

jasagredo · 2025-06-30T11:01:09Z

+            }
+  forM_ snapshotSlots $ \slot -> do
+    -- Prune the 'DbChangelog' such that the resulting anchor state has slot
+    -- number @slot@.


Suggested change

-- number @slot@.

-- number @slot@ or younger.

?

Each s in snapshotSlots is the slot of a ledger state in DbChangelog (see the contract of onDiskSnapshotSelector), so it is true as written. (Otherwise, we wouldn't take a snapshot for the requested slot, so snapshots wouldn't be predictable.)

jasagredo · 2025-09-02T10:34:58Z

+-- of 'sfaInterval'.
+data SnapshotFrequencyArgs = SnapshotFrequencyArgs
+  { sfaInterval :: OverrideOrDefault SlotNo
+  -- ^ Try to write snapshots every 'sfaInterval' many slots. Must be positive.


Isn't it necessarily positive by using a SlotNo?

It is necessarily non-negative, but not necessarily positive 😅

I changed this to NonZero Word64 (as SlotNos are usually used to signify absolute slot numbers, not distances between slots; we do the same in the HFC).

jasagredo · 2025-09-02T10:37:58Z

+  let immutableStates =
+        AS.dropNewest (fromIntegral (envMaxRollbacks env)) $ changelogStates chlog
+      immutableSlots :: [SlotNo] =
+        nubOrd . mapMaybe (withOriginToMaybe . getTipSlot) $


This nubOrd I imagine is there only because of EBBs, right?

Exactly, added a comment

jasagredo · 2025-09-02T10:40:44Z

+    atomically $ modifyTVar (ldbChangelog env) (prune pruneStrat)
+    -- Flush the LedgerDB such that we can take a snapshot for the new anchor
+    -- state due to the previous prune.
+    withWriteLock
+      (ldbLock env)
+      (flushLedgerDB (ldbChangelog env) (ldbBackingStore env))


I'm confused. I seem to remeber the flow was the opposite, first flush then prune, no?

EDIT: Ah now I think pruning just affects the changelog states and not the diffs.

Ah now I think pruning just affects the changelog states and not the diffs.

Exactly 👍

jasagredo · 2025-09-02T10:42:26Z

+          (configCodec . getExtLedgerCfg . ledgerDbCfg $ ldbCfg env)
+          (LedgerDBSnapshotEvent >$< ldbTracer env)
+          (ldbHasFS env)
+          (anchorHandle $ snd $ prune pruneStrat lseq)


Here we prune the lseq that we provide to the function that takes the snapshot, but we do not modify the ldbSeq. However from what I understood about V1 above, there we do prune the dbChangelog in the environment. Why this discrepancy?

Above we do:

atomically $ modifyTVar (ldbChangelog env) (prune pruneStrat)

Good call pointing that out. Morally, I think the V2 semantics are preferrable: taking a snapshot should not modify the stored states. However, with V1, this is not possible: We can only perform a snapshot for the last flushed state, so we have no choice there.

Note that there already are some differences between V1 and V2 before this PR due to this: With V1, we create snapshots for the last flushed state, whereas with V2, we create snapshots for the immutable tip.

Given that V1 is going to go away "soon", I am inclined to not worry about this too much, but maybe that is too optimistic.

geo2a · 2026-04-15T12:15:27Z

-      now <- getMonotonicTime
-      pure $ now `diffTime` lastWrite
-  RAWLock.withReadAccess (ldbOpenHandlesLock env) $ \() -> do
+implTryTakeSnapshot snapManager env snapshotRequestTime getRandomDelay = do


Calculate snapshotRequestTime inside this function using now, as it was before.

Superseded by the rework of the snapshot policy for predictable snapshots, with dedicated new tests LedgerDB: implement predictable snapshotting

It is no longer needed by the predictable snapshotting logic.

- tryTakeSnapshot: now accepts a `Time` argument. The argument specifies the time at which the snapshot should be taken - LedgerDBEnv: rename ldbLastSnapshotWrite to ldbLastSnapshotRequestedAt. Track the request time rather than the time a snapshot actually finished. - implTryTakeSnapshot: add a `delay` argument. How long should we block before actually taking the snapshot after determining the slots to snapshot - add `cdbSnapshotDelayRNG` and use this to determine how long we should wait before taking a snapshot - add orphan `NoThunks` `StdGen` instance - add `cdbsSnapshotDelayRNG to `ChainDbSpecificArgs` - add `onDiskSnapshotDelayRange` to `SnapshotPolicy` to allow configuration of the delay between a snapshot being requested and being taken - add LedgerDB snapshot delay trace events. Use these events in the test suite to ensure we don't add blocks while snapshots are occurring (and therefore make an accurate number of snapshots). - add a test ensuring that blocks can be added while a snapshot is enqueued - ledgerDbMaintenaceThread -> ledgerDbMaintenanceThread

This commit brings back the fix from #1814, which synchronises the process of taking a ledger state snapshot and copying of blocks into the immutable DB.

The value of this variable should be the time when the snapshot was requested, not finished taking.

amesgen requested review from dnadales, fraser-iohk, geo2a, jasagredo and nfrisby as code owners June 30, 2025 08:32

amesgen added this to Consensus Team Backlog Jun 30, 2025

amesgen moved this to 👀 In review in Consensus Team Backlog Jun 30, 2025

amesgen self-assigned this Jun 30, 2025

amesgen force-pushed the amesgen/predictable-snapshots branch from 0e21795 to 4d5bb72 Compare June 30, 2025 08:52

jasagredo reviewed Jun 30, 2025

View reviewed changes

amesgen force-pushed the amesgen/ledgerdb-garbage-collect-states branch from a8fa7e2 to b503dc3 Compare June 30, 2025 11:52

amesgen force-pushed the amesgen/predictable-snapshots branch from 4d5bb72 to 6f063b3 Compare June 30, 2025 11:52

amesgen force-pushed the amesgen/ledgerdb-garbage-collect-states branch from b503dc3 to 6c78fad Compare July 2, 2025 08:14

amesgen force-pushed the amesgen/predictable-snapshots branch from 6f063b3 to dab2fbf Compare July 2, 2025 08:19

amesgen force-pushed the amesgen/ledgerdb-garbage-collect-states branch from 6c78fad to 48bb1fe Compare July 7, 2025 08:26

amesgen force-pushed the amesgen/predictable-snapshots branch from dab2fbf to 70fcdc3 Compare July 7, 2025 08:27

amesgen force-pushed the amesgen/ledgerdb-garbage-collect-states branch from 48bb1fe to ad7acfa Compare July 9, 2025 12:26

amesgen force-pushed the amesgen/predictable-snapshots branch from 70fcdc3 to 2ecc9b9 Compare July 9, 2025 12:26

amesgen mentioned this pull request Jul 17, 2025

Create occasional snapshots while replaying #1596

Open

amesgen force-pushed the amesgen/ledgerdb-garbage-collect-states branch from ad7acfa to d791dfb Compare August 4, 2025 11:26

amesgen force-pushed the amesgen/predictable-snapshots branch from 2ecc9b9 to 9a0585f Compare August 4, 2025 11:27

amesgen force-pushed the amesgen/ledgerdb-garbage-collect-states branch from d791dfb to 247c489 Compare August 8, 2025 14:13

amesgen force-pushed the amesgen/predictable-snapshots branch from 9a0585f to 5fdc096 Compare August 8, 2025 14:13

amesgen force-pushed the amesgen/ledgerdb-garbage-collect-states branch from 247c489 to 08bda65 Compare August 13, 2025 14:41

amesgen force-pushed the amesgen/predictable-snapshots branch from 5fdc096 to 2e8e7be Compare August 13, 2025 14:42

amesgen force-pushed the amesgen/ledgerdb-garbage-collect-states branch from 08bda65 to dec284f Compare August 13, 2025 15:27

amesgen force-pushed the amesgen/predictable-snapshots branch from 2e8e7be to e542170 Compare August 13, 2025 15:28

jasagredo reviewed Sep 2, 2025

View reviewed changes

Base automatically changed from amesgen/ledgerdb-garbage-collect-states to main September 2, 2025 14:46

geo2a force-pushed the amesgen/predictable-snapshots branch from 6258cef to 74d1d62 Compare April 13, 2026 15:09

dnadales mentioned this pull request Apr 14, 2026

Cardano Node Ledger State Certification - PoC input-output-hk/mithril#2525

Open

10 tasks

geo2a linked an issue Apr 14, 2026 that may be closed by this pull request

Optional random delay when creating snapshots #1573

Open

geo2a force-pushed the amesgen/predictable-snapshots branch 2 times, most recently from 096896d to 7be6ea5 Compare April 15, 2026 11:16

geo2a removed a link to an issue Apr 15, 2026

Enrich ChainDB state machine model with ledger snapshots #585

Open

geo2a force-pushed the amesgen/predictable-snapshots branch from 7be6ea5 to f6f2731 Compare April 15, 2026 11:18

geo2a reviewed Apr 15, 2026

View reviewed changes

Comment thread changelog.d/20260407_141821_georgy.lukyanov_predictable_snapshots.md Outdated

geo2a reviewed Apr 15, 2026

View reviewed changes

geo2a force-pushed the amesgen/predictable-snapshots branch 7 times, most recently from 826f567 to d181bb5 Compare April 17, 2026 11:38

amesgen and others added 13 commits April 17, 2026 13:41

Remove LedgerDB.SnapshotPolicy test

c5e9f0f

Superseded by the rework of the snapshot policy for predictable snapshots, with dedicated new tests LedgerDB: implement predictable snapshotting

LedgerDB: implement predictable snapshotting

d1c894a

LedgerDB: remove replayed blocks counter

e4eaea9

It is no longer needed by the predictable snapshotting logic.

Add ChainDB test for ledger snapshots

86bf48c

Flush immutable blocks before taking a ledger state snapshot.

f7a2d05

This commit brings back the fix from #1814, which synchronises the process of taking a ledger state snapshot and copying of blocks into the immutable DB.

Trace slots of delayed snapshots

6021dbf

ChainDB q-s-m: test the interaction of VolatileDB and snapshots

9a42a93

ChainDB: comment on interaction of tryTakeSnapshot and garbageCollect

3cb3e5d

Remove Time argument to tryTakeSnapshot

bffbd5d

ChainDB q-s-m: do not copy blocks to ImmutableDB on snapshot

a358a71

LedgerDB V1: fix the update of ldbLastSnapshotRequestedAt

ea50099

The value of this variable should be the time when the snapshot was requested, not finished taking.

LedgerDB: close handles after taking snapshots

6c2bc63

geo2a force-pushed the amesgen/predictable-snapshots branch from d181bb5 to 6c2bc63 Compare April 17, 2026 11:42

Conversation

amesgen commented Jun 30, 2025 • edited by geo2a Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amesgen commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

amesgen commented Jun 30, 2025 •

edited by geo2a

Loading

amesgen commented Jun 30, 2025 •

edited

Loading