feat(data): add partial vector data structure

[vapourismo]

Part of RV-861

What

Introduces a partial vector data structure that lets you define arbitrary parts of a large space (0..=usize::MAX).

Why

I found this data structure to be very useful in implementing the byte array state component (#732). It was also handy in making DataSpace more space efficient during proof generation (#736).

How

I added lots of commentary within the data structure module. Please let me know what you think is missing.

Manually Testing

make all

Tasks for the Author

• Link all Linear issues related to this MR using magic words (e.g. part of, relates to, closes).
• Eliminate dead code and other spurious artefacts introduced in your changes.
• Document new public functions, methods and types.
• Make sure the documentation for updated functions, methods, and types is correct.
• Add tests for bugs that have been fixed.
• Explain changes to regression test captures when applicable.
• Write commit messages in agreement with our guidelines.
• Self-review your changes to ensure they are high-quality.
• Complete all of the above before assigning this MR to reviewers.

[vapourismo]

• feat(data): add partial vector data structure #731 : 2 dependent PRs (#732 , #736 ) 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.36%. Comparing base (94b0af2) to head (6e252c0).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #731      +/-   ##
==========================================
+ Coverage   91.26%   91.36%   +0.09%     
==========================================
  Files         108      109       +1     
  Lines       20232    20419     +187     
  Branches    20232    20419     +187     
==========================================
+ Hits        18465    18655     +190     
+ Misses       1391     1388       -3     
  Partials      376      376              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Benchmark results for revision 4e72418:

Metric	Duration	TPS
Mean	1.521256038s	26.294
Worst	1.535359677s	26.053
Best	1.512220247s	26.451
Standard Deviation	±4.881965ms	±0.084

Full results

Run	Transfers	Duration	TPS
1	40	1.521511074s	26.290
2	40	1.520791297s	26.302
3	40	1.52394236s	26.248
4	40	1.51809757s	26.349
5	40	1.529806267s	26.147
6	40	1.512220247s	26.451
7	40	1.535359677s	26.053
8	40	1.518627643s	26.340
9	40	1.523144174s	26.261
10	40	1.518803022s	26.337
11	40	1.519169058s	26.330
12	40	1.526341399s	26.206
13	40	1.515087724s	26.401
14	40	1.521600506s	26.288
15	40	1.518926563s	26.334
16	40	1.520690715s	26.304
17	40	1.522908422s	26.266
18	40	1.518589446s	26.340
19	40	1.520274896s	26.311
20	40	1.51922869s	26.329

Compare the results above with those for the default branch.

[thomasathorne]

This generally looks great, the PR was fun to review!

My one concern is that there is nothing keeping the tree balanced. So for example

#[test]
fn unbalanced_tree() {
    let mut vec: PartialVec<u8> = PartialVec::undefined(0);

    for i in 0..40000 {
        vec.define(i * 2 + 1, vec![0]);
    }
}

is enough to fail with a stack overflow.

That is obviously a long way from the intended use case, but 40,000 entries isn't that many. I'd argue we should either consider fixing this or be very careful not to use this in any situations where an adversary can choose the shape of the data.

[emturner]

overall makes sense, main comment is that I'd argue a significant number of edge cases seem to exist w.r.t. integer overflow that should be guarded against

[vapourismo]

This generally looks great, the PR was fun to review!

My one concern is that there is nothing keeping the tree balanced. So for example

#[test]
fn unbalanced_tree() {
    let mut vec: PartialVec<u8> = PartialVec::undefined(0);

    for i in 0..40000 {
        vec.define(i * 2 + 1, vec![0]);
    }
}

is enough to fail with a stack overflow.

That is obviously a long way from the intended use case, but 40,000 entries isn't that many. I'd argue we should either consider fixing this or be very careful not to use this in any situations where an adversary can choose the shape of the data.

40k operations is definitely more than it is currently designed to handle.

I'll have a look into how we can avoid the fragmentation a bit more.

[vapourismo]

This generally looks great, the PR was fun to review!
My one concern is that there is nothing keeping the tree balanced. So for example

#[test]
fn unbalanced_tree() {
    let mut vec: PartialVec<u8> = PartialVec::undefined(0);

    for i in 0..40000 {
        vec.define(i * 2 + 1, vec![0]);
    }
}

is enough to fail with a stack overflow.
That is obviously a long way from the intended use case, but 40,000 entries isn't that many. I'd argue we should either consider fixing this or be very careful not to use this in any situations where an adversary can choose the shape of the data.

40k operations is definitely more than it is currently designed to handle.

I'll have a look into how we can avoid the fragmentation a bit more.

@thomasathorne

So the stack overflow comes from the destructor 😕

I think I found a way to flatten the structure such that those attacks aren't possible. There are some trade-offs, though. I will create a new PR to stack on this one for the improvement.

[vapourismo]

I have created #747 to improve on the stack overflow issue that @thomasathorne has raised.

[thomasathorne]

Happy to approve this now and look forward to the follow-up.