bug: data race in follower cache Peek() allows concurrent herocache reads

[peterargue]

Problem

Running TestFollowerHappyPath with the race detector reports a data race:

WARNING: DATA RACE
Read at 0x00c000550aa0 by goroutine N:
  herocache/backdata.(*Cache).linkedValueOf()       ← cache.go:473
  engine/common/follower/cache.(*Cache).Peek()
  engine/common/follower.(*ComplianceCore).OnBlockRange()
  engine/common/follower.(*ComplianceEngine).processConnectedBatch()

Previous write at 0x00c000550aa0 by goroutine M:
  herocache/backdata.(*Cache).linkedValueOf()       ← cache.go:485
  engine/common/follower/cache.(*Cache).Peek()
  engine/common/follower.(*ComplianceCore).OnBlockRange()
  engine/common/follower.(*ComplianceEngine).processConnectedBatch()

Root Cause

cache.Peek() in engine/common/follower/cache/cache.go acquires only a read lock (RLock) before calling c.backend.Get(). However, herocache.Cache.Get() is not safe for concurrent reads — in linkedValueOf() it performs a lazy write (c.buckets[b].slots[s].slotAge = slotAgeUnallocated) when it detects a stale slot.

With defaultBatchProcessingWorkers = 4, all four processConnectedBatch goroutines can call Peek() concurrently. All four hold the read lock simultaneously and their concurrent calls to herocache.Get() race on that slotAge write, corrupting the cache's internal state.

Fix

Change Peek() to use an exclusive lock instead of a shared lock:

func (c *Cache) Peek(blockID flow.Identifier) *flow.Proposal {
    c.lock.Lock()
    defer c.lock.Unlock()
    ...
}

All other methods on Cache already use the exclusive lock. Peek() was the only outlier.

This issue was identified in collaboration with Claude.