inclusionChecker: rare possibility of getting ahead of submissions

[ezekiel]

We have encountered the very rare occasion where the inclusionChecker is following very closely with the log/shard tree-head and happens to miss an included certificate. Re-starting ct-woodpecker with the LogIndex set back a couple tens of thousands allows ct-woodpecker to "see" the certificate it missed, and then continue on as normal.

I don't yet know how it could have missed the certificate it submitted, only that ct-woodpecker at the time was very up-to-date, following the shard's tree-head quite closely (ct-woodpecker itself may have been the only entity submitting to this shard).

[cpu]

Thanks for the bug report.

Can you share the log output from the time period before you changed the LogIndex and restarted ct-woodpecker?

[ezekiel]

Attaching a 2-hour timeframe for certSubmitter and inclusionChecker logs only. Looks like the overlooked cert was submitted 02:04:22.
ct-woodpecker.log

[cpu]

@ezekiel Thanks, that's perfect 👍