fix: snapshot workspace before parallel grading to prevent race condition

[ScuttleBot]

Problem

When parallel grading is enabled (ThreadPoolExecutor, introduced in #351), grade_task() runs in a background thread while the main thread immediately moves on to the next task and calls prepare_task_workspace(), which rebuilds the shared agent workspace directory. The background grader then reads the next task's workspace files instead of the current one's, causing:

• Transcript mismatches (task N's transcript archived as task N+1's)
• Incorrect automated scores (task_sanity scoring 0 because its workspace state is gone)

Fix

Before submitting grade_task to the ThreadPoolExecutor, snapshot the workspace directory into an isolated temp directory via shutil.copytree. A shallow copy of execution_result with the snapshot path replaces the live workspace reference, so the background grader reads stable files regardless of what the main thread does next.

Cleanup happens in _wait_for_pending_grade() after grading completes (shutil.rmtree).

The synchronous grading path is unchanged.

Changes

• Added shutil and tempfile imports
• Added _snapshot_workspace_for_grading logic inline in the parallel submission block
• Added pending_grade_snapshot_dir tracking variable
• Added cleanup in _wait_for_pending_grade()

Closes #365

[kilo-code-bot]

WARNING: shutil.copytree is not wrapped in a try/except here. If it raises (e.g. disk full, broken symlink, permission error), the exception propagates uncaught through the task loop and crashes the entire benchmark run — losing all results collected so far. The temp dir created on the line above would also be leaked.

Consider wrapping the snapshot block in a try/except that falls back to using the original result (accepting the small race-condition risk) rather than aborting the run:

try:
    snapshot_dir = tempfile.mkdtemp(prefix="pinchbench_grade_")
    shutil.copytree(workspace_path, os.path.join(snapshot_dir, "ws"), dirs_exist_ok=True)
    snapshot_workspace = os.path.join(snapshot_dir, "ws")
    grade_result_copy = dict(result)
    grade_result_copy["workspace"] = snapshot_workspace
    grade_kwargs["execution_result"] = grade_result_copy
    pending_grade_snapshot_dir = snapshot_dir
except Exception as exc:
    logger.warning("Workspace snapshot failed, grading with live path: %s", exc)
    pending_grade_snapshot_dir = None

[kilo-code-bot]

Code Review Summary

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity	Count
WARNING	1

Issue Details (click to expand)

WARNING

File	Line	Issue
scripts/benchmark.py	976	shutil.copytree unhandled exception can crash the entire benchmark run and leak the temp dir

Files Reviewed (2 files)

• README.md — no issues (task count update)
• scripts/benchmark.py — 1 issue

---

The race-condition fix is well-reasoned and correctly scoped. The snapshot + cleanup pattern is clean. The only concern is the missing error guard around shutil.copytree — a disk-full or permission error would propagate uncaught and abort the whole run. See the inline comment for a suggested fallback.

Fix these issues in Kilo Cloud

---

_{Reviewed by claude-4.6-sonnet-20260217 · 159,069 tokens}

[ScuttleBot]

Closing in favor of #372 which has a cleaner implementation (dedicated helper function, better documentation, no unrelated README changes). Both PRs solved the same race condition the same way.