Skip to content

refactor: share checkpoint read planning across scan metadata paths#2459

Open
ethan-tyler wants to merge 6 commits intodelta-io:mainfrom
ethan-tyler:refactor/checkpoint-read-planning
Open

refactor: share checkpoint read planning across scan metadata paths#2459
ethan-tyler wants to merge 6 commits intodelta-io:mainfrom
ethan-tyler:refactor/checkpoint-read-planning

Conversation

@ethan-tyler
Copy link
Copy Markdown
Contributor

scan_metadata() and parallel_scan_metadata() had separate checkpoint planning logic. The parallel path used a conservative hard coded CheckpointReadInfo and could miss projected checkpoint capabilities (like parsed stats columns) that the normal path already understood.

This fix here is to centralize projected checkpoint planning in LogSegment, let CheckpointManifestReader accept a caller planned schema, and thread it through the sequential phase allowing both paths to use the same checkpoint knowledge.

For single part checkpoints, the parallel path defers finalization until the sequential phase knows which files it's handing off, then updates the replay processor before parallel processing begins.

This is an internal only change

Tests cover stats_parsed preservation and partition_values_parsed planning for single part checkpoints in the parallel path.

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 24, 2026

Codecov Report

❌ Patch coverage is 87.30159% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.38%. Comparing base (ac9dc19) to head (d74275b).

Files with missing lines Patch % Lines
kernel/src/log_segment/mod.rs 84.61% 17 Missing and 5 partials ⚠️
kernel/src/parallel/sequential_phase.rs 82.66% 1 Missing and 12 partials ⚠️
kernel/src/parallel/parallel_scan_metadata.rs 89.65% 5 Missing and 1 partial ⚠️
kernel/src/scan/mod.rs 90.47% 3 Missing and 1 partial ⚠️
kernel/src/log_reader/checkpoint_manifest.rs 93.33% 0 Missing and 2 partials ⚠️
kernel/src/scan/log_replay.rs 96.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2459      +/-   ##
==========================================
- Coverage   88.39%   88.38%   -0.01%     
==========================================
  Files         175      175              
  Lines       58121    58421     +300     
  Branches    58121    58421     +300     
==========================================
+ Hits        51375    51637     +262     
- Misses       4784     4804      +20     
- Partials     1962     1980      +18     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions github-actions Bot added the breaking-change Public API change that could cause downstream compilation failures. Requires a major version bump. label Apr 24, 2026
Signed-off-by: Ethan Urbanski <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change Public API change that could cause downstream compilation failures. Requires a major version bump.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants