feat(rust/core): add async traits for Driver, Database, Connection, Statement

[if0ne]

Hi, I’m back again. This is a continuation of this PR: #3712

I’ve returned with a draft of the asynchronous traits. They fully mirror the synchronous API (with a couple of differences in input arguments), adding Future + Send as the return type.

As further development, I can propose:

• Abstract away from async executors and introduce an AsyncExecutor trait with a single block_on method. In adbc_core, add helper structures like SyncDriverWrapper<E: AsyncExecutor, D: AsyncDriver> and others, and implement the corresponding synchronous traits for them.

• Add some procedural macro magic to automatically implement the synchronous traits by generating the code described above.

[if0ne]

What do you think? @lidavidm and @eitsupi

[lidavidm]

CC @felipecrv @mbrobbel too

[mbrobbel]

For the Send bound stuff I would suggest using https://docs.rs/trait-variant/0.1.2/trait_variant/ (as suggested in https://blog.rust-lang.org/2023/12/21/async-fn-rpit-in-traits/#async-fn-in-public-traits).

For async vs sync I would suggest using the async trait as the default and provided sync helper traits and structs. But the truth is; there is no elegant solution available today for this.

[if0ne]

trait_variant will help solve the Send bound problem, but it introduces another problem: the macro does not add additional Send bound to function input arguments if they are of an opaque type. Should we use explicit types instead?

[felipecrv]

Once the code in blocking is moved back to the current location I can give another review.

@lidavidm do we really need everything to be a Future, even get/set options?

[if0ne]

Once the code in blocking is moved back to the current location I can give another review.

@lidavidm do we really need everything to be a Future, even get/set options?

I made the AsyncOptionable because in the DataFusion implementation, setting options involved asynchronous operations. It might be possible to make get_option_* sync, but I would keep set_option async.

https://github.com/if0ne/arrow-adbc/blob/61963f8e450926da1657f31112db9498f789b380/rust/driver/datafusion/src/lib.rs#L247

[lidavidm]

Even get_option may involve I/O in some cases (e.g. if the driver has to run a query to fetch some information), unfortunately

[lidavidm]

Does arrow-rs not already have a trait for this?

[lidavidm]

I see that arrow-rs does not implement ArrowAsyncDeviceStreamHandler etc yet. I think we need to figure this out in arrow-rs

[if0ne]

Related issue apache/arrow-rs#7228

[lidavidm]

Filed apache/arrow-rs#9166

[if0ne]

After the merge, I’ll get back to this PR

[lidavidm]

Merged (at least the trait). We can bump to the next version when available (possibly via feature? Or maybe splitting async traits into a new crate?)

[if0ne]

I will split the current PR into three pull requests:

1. Moving the existing structures into a separate module
2. Adding sendable async traits as an async feature
3. Adding bridge structures, which will probably also be under the async feature

[lidavidm]

I think there needs to be explanation somewhere of what "local" means (traits that are not Send?)

Additionally, does it make sense to have non-Send variants? It's not clear to me that that's useful (when would a connection be limited to a single thread?)

[if0ne]

This is about flexibility. I think that, in practical terms, most drivers will implement the async trait with a Send bound. But someone might want to implement an async Rust driver that uses an FFI library to connect to a specialized database.

[lidavidm]

Is it actually worth optimizing for that flexibility, though? (Versus, say, expecting that such a very specialized database would instead manage this internally?)

[lidavidm]

Because as it stands, this means duplicating every method/trait

[if0ne]

IMO it’s not worth supporting the non-Send variant (maybe someday in the future, when the crate stabilizes and reaches versions > 1.0.0).

Personally, I was planning to use adbc for the wasm target, and as far as I understand, all the necessary structures are Send.

[eitsupi]

FYI: I found another driver (which seems to be fairly recent): https://github.com/marconae/exarrow-rs

[paleolimbot]

The sedona-db implementation is unused to my knowledge so don't worry about any compatibility issues there!

[lidavidm]

@marconae sorry for the random ping, but would you like to weigh in on async ADBC interfaces? 🙂

[felipecrv]

This is about flexibility.

We might end up offering too much flexibility in a standard that is supposed to make database access more uniform. If a database doesn't allow its handles to move between threads, it's the responsibility of the ADBC driver wrapping these database APIs to fix for that rather than bubbling up these restrictions to Rust users of ADBC via the type system which basically means a rewrite of the entire application depending on the ADBC driver that it uses.

[marconae]

@lidavidm thx for including me. I had to go async, because I am fetching data from an async web-socket protocol. Maybe the my pattern is a bit unusual. For me it was fine to work against a synchronous library and wrap it.