update datafusion to 52 & update deps

[hozan23]

No description provided.

[hozan23]

Since datafusion removed the pyarrow feature, we should migrate to datafusion-python. However, version 52 of datafusion-python has not yet been published, which is why we encountered the previous issue with CI. Let's wait for it.

[hozan23]

Updates:
datafusion-python merged the preparation branch to version 52, but it hasn't been released yet. I am following the release, waiting for version 52 release.

[nuno-faria]

@hozan23 datafusion-python has been updated: https://crates.io/crates/datafusion-python/52.0.0

[hozan23]

@nuno-faria @phillipleblanc Can you have a final review please before merging to main?

[nuno-faria]

Thanks @hozan23, overall LGTM but there appears to be an error when executing the integration tests, namely at test_flight_sql_data_source. The df.count() triggers a panic in arrow:

thread 'flight::test_flight_sql_data_source' panicked at .../arrow-select-57.3.0/src/coalesce.rs:462:9:
assertion `left == right` failed
  left: 2
 right: 0

According to this comment, it appears that it was not supposed to work before in the first place. As a workaround, we can convert the df.count() into a df.collect().await?.iter().map(|b| b.num_rows()).sum::<usize>().

[hozan23]

Hi @nuno-faria
I have fixed the df.count() issue. I also attempted to add integration tests to the CI, but i think it is non-trivial to add them, I reverted those changes. I have kept the fix for the df.count() issue you mentioned.

Let me know if it looks good to you so we can merge it.

[nuno-faria]

LGTM, thanks again @hozan23.