Add support for multiple snapshot intervals to SnapshotPolicy

[fraser-iohk]

At the moment, the SnapshotPolicy has a single snapshot interval which determines the period between slots for which snapshots should be taken. To support Mithril, we need to be able to take both predictable snapshots and the existing periodic snapshots within the same node.

In order to support this, we need to allow node operators to configure their nodes with multiple snapshot intervals and ensure that the intervals are treated independently within the snapshotting mechanism.

[fraser-iohk]

Option 1: Multiple intervals, single SnapshotPolicy

Keep everything the same except that sfaInterval is a non-empty list of Word64s

Pros

• Code change is easy (set of slot numbers for all intervals is the union of slot numbers for each interval)

Cons

• Likely to miss snapshots in conjunction with the snapshot policy's rate limiting mechanism
• Can't guarantee Mithril snapshots will be taken
• No way to differentiate between snapshot series on disk (this might not be an issue)

Option 2: Multiple separate SnapshotPolicyArgs values

Allow the node operator to specify multiple snapshot frequencies in the configuration, and use one for the existing periodic snapshot mechanism, and another for the per-epoch Mithril snapshots

Pros

• Rate limits / delays / maximum number of snapshots can be per-policy

Cons

• More effort involved to change the code, especially w.r.t. matching snapshots with their on-disk counterparts
• "last snapshot taken at" time would need to be tracked for each of the snapshot policies
• Rate-limiting is still an issue, since the node could still degrade due to consecutive snapshots

Option 3: Named snapshots

This is effectively option 2 but with explicitly named snapshot policies rather than a list, and naming those snapshot series on disk

Pros

• Allows for on-disk snapshots to be labelled for clarity, easier to track on-disk snapshots, easier to add / remove / adjust snapshot policies and have them remain stable

Cons

• A lot of code changes required, both in consensus and node
• Not strictly backwards compatible with the existing config format
• Snapshots need to be named

Questions

• Do we need to ensure that taking Mithril snapshots is guaranteed? If so, how can we avoid a situation where the node performance degrades due to multiple snapshots being taken in succession?
• Do we need snapshots for Mithril to be easily distinguishable on disk?

[fraser-iohk]

If we're okay with not supporting both periodic and Mithril snapshotting simultaneously, we don't need to make any changes to the snapshot policy handling in consensus at all -- node operators can continue using their existing configuration (which will switch from being time-based to slot-based, but should be effectively the same), or can change their configuration to support Mithril's snapshotting requirement

The existing rate-limit mechanism should be documented in the code, though, and we should add a startup sanity check to consensus to warn users whose snapshot frequency / rate limit is weird enough that it may cause performance issues in the node