Storehouse: Background indexing #8244
Description
Problem Definition
Enabling Storehouse on an execution node requires indexing the root checkpoint file along with all trie updates for every executed block. This process currently introduces downtime—approximately 2–3 hours on mainnet. If the checkpoint file is outdated, the execution node needs additional time to catch up and execute the latest block.
Solution
To reduce or eliminate this downtime, we can introduce a background indexing process that runs while the node continues executing blocks without Storehouse enabled. Once the background indexer has fully caught up, we can switch the node to use Storehouse seamlessly, with no downtime.
An additional benefit of this approach is the ability to periodically validate data consistency without enabling Storehouse. Since the data is already indexed in the background, we can run regular consistency checks to ensure that all payloads stored in Storehouse match those in the checkpoint file.
TODO
- Introduce a flag to enable a background indexer that processes the checkpoint file and trie updates for each executed and sealed block.
- Implement a full consistency validation between Storehouse data and the checkpoint file.