Block data mover design

[Lyndon-Li]

Add block data mover design for block level incremental backup by integrating with Kubernetes CBT

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.64%. Comparing base (556d582) to head (3218944).
⚠️ Report is 36 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9528      +/-   ##
==========================================
+ Coverage   60.54%   60.64%   +0.10%     
==========================================
  Files         386      387       +1     
  Lines       36370    36518     +148     
==========================================
+ Hits        22020    22148     +128     
- Misses      12770    12782      +12     
- Partials     1580     1588       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kaovilai]

The current code deletes the VolumeSnapshot and VolumeSnapshotContent Kubernetes objects but retains the underlying storage snapshot. This means:

• The CSI snapshot handle is only stored in DataUpload.Status.SnapshotID
• There's no VolumeSnapshotContent object in the cluster pointing to the retained snapshot
• To use it as base_snapshot_id for CBT, the next backup needs to retrieve this handle from the Kopia repository metadata
• If incrementalBackup5, 4, 3, 2, 1 all the way to the original initial full backup is deleted, will velero recreate VolumeSnapshotContent from each base_snapshot_id with RetentionPolicy: Delete, in order to trigger deletion in the underlying storage?

[Lyndon-Li]

The current code deletes the VolumeSnapshot and VolumeSnapshotContent Kubernetes objects but retains the underlying storage snapshot. This means:

This is NOT true for either the existing file system data mover and the new block data mover. Both the VS/VSC and the underlying storage snapshot will be deleted after backup.

• The CSI snapshot handle is only stored in DataUpload.Status.SnapshotID

If SnapshotID contains the ChangeID (this is the fact for many storages), then the answer is NO, as you see in the design doc, the ChangeID is also saved in the backup repository.

VolumeSnapshotContent from each base_snapshot_id with RetentionPolicy: Delete, in order to trigger deletion in the underlying storage?

As mentioned above, the backend storage snapshot is deleted at the first place when the backup completes.

[Lyndon-Li]

@shubham-pampattiwar @kaovilai
I have updated the design to address the questions during the community meeting. Let me know if I missed anything.

[kaovilai]

As mentioned above, the backend storage snapshot is deleted at the first place when the backup completes.

Have you tested if you delete backend storage snapshot when backup completes if GetMetadataDelta(base_snapshot_id,next) would work if base_snapshot_id is already deleted?

My understanding is GetMetadataDelta(base_snapshot_id, target_snapshot_id) needs both CSI snapshot IDs to be valid at the storage backend. The CSI driver must be able to compute the block-level diff between them. If the base snapshot has been deleted from the storage backend, this call fails.

[kaovilai]

the backup application ... take that into consideration in its snapshot retention policy ... should save the CSI snapshot handle somewhere for future use if it deletes the VolumeSnapshot object immediately after it has performed a backup.

[Lyndon-Li]

@kaovilai

As mentioned above, the backend storage snapshot is deleted at the first place when the backup completes.

Have you tested if you delete backend storage snapshot when backup completes if GetMetadataDelta(base_snapshot_id,next) would work if base_snapshot_id is already deleted?

My understanding is GetMetadataDelta(base_snapshot_id, target_snapshot_id) needs both CSI snapshot IDs to be valid at the storage backend. The CSI driver must be able to compute the block-level diff between them. If the base snapshot has been deleted from the storage backend, this call fails.

That varies from storages. I have added a section about Volume Snapshot Retention in the design. But I don't think implementing the snapshot retention is a high priority, let's discuss it tomorrow.