Zero copy data API

[brancz]

Let me just start with, we're not currently a user of uPlot, but we're evaluating various options.

Zero-copy is important to us because our system often returns 20MB+ data as Apache Arrow records, and expanding them to regular objects/arrays in JS is not workable, both from a latency perspective and memory (expanding them can easily yield 100MB+ data, which is also not something we want to burden clients with).

I'm not very familiar with uPlot's architecture, so I don't know if it's feasible to do something like this. That said, is there an appetite for a zero-copy data API? If so, what would it take to do this?

[leeoniya]

i'm aware of https://github.com/uwdata/flechette. my assumption is that zero-copy would require using library-specific iterator apis/methods to extract each value on demand to determine x and y scale limits, as well as again at render time. this might be manageable overhead when rendering a subset of the data, but i imagine it would.be quite expensive when rendering the entire dataset.

this is all conjecture that has to be validated by testing. uplot doesnt have an api for defining a custom data format with custom iterators because almost universally this destroys performance. uPlot does support rendering typed arrays (though there is no support for null-valued explicit gaps with these).

if you plan to render the entire dataset, i would suggest taking your data and using fletchette to read each x and y value twice (as two consecutive loops). then compare that to the cost of extraction into js-native arrays plus the same two loops over the js-native arrays.

if you are able to attach a representative dataset here, i can try to help with such an assessment.

[leeoniya]

oh, there is also https://github.com/hyparam/hyparquet, not sure if that is helpful. seems more like something for on-demand / incremental data fetching over http

[leeoniya]

also,

https://github.com/kylebarron/parquet-wasm might be useful, but https://duckdb.org/docs/stable/clients/wasm/overview might NOT be useful

[leeoniya]

the ideal zero-copy scenario might be a WASM arrow/parquet reader (running in a web worker) with the ability to expose arrow columns as typed arrays or buffers that can be transferred to the main thread via https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Transferable_objects

if this is something you can get working, you should hopefully be able to leverage uPlot with minimal changes. i can add a toggle for supporting explicit data gaps as NaN values in the data rather than null values in the data to bring better parity for typed arrays.

[brancz]

Interesting pointers, thank you! We'll play around with these and see!

[leeoniya]

something i should mention is that you should avoid BigInt since it will not interop with normal js code / math. also timestamps have to be in epoch milliseconds and the smallest resolution is 1ms to accommodate javascript Date objects. if you need micro or nano timestamps the effort to make that work in uPlot will be non-trivial and unlikely to happen until the Temporal web api is well established.