[FEA] [Multi-GPU Polars] Avoid GPU Requirement on Client During IR Lowering

[madsbk]

Avoid GPU Requirement on Client During IR Lowering

Summary

The current design may require a GPU on the client during the IR lowering stage. Ideally, the client should not require a GPU.

Problem

IR lowering (lower_ir_graph) is currently performed on the client, which can trigger GPU usage. This introduces an implicit requirement that the client machine has a GPU, even in distributed setups where all actual execution happens on workers.

Proposed Direction

Move IR lowering from the client to the workers.

Challenges

A naive approach, where each worker independently runs lower_ir_graph, would lead to redundant work:

• All workers would sample the same Parquet metadata and row groups.
• This is inefficient and may lead to unnecessary overhead.

cc. @rjzamora, @TomAugspurger

[TomAugspurger]

Overall, I think I'm OK with multiple workers (redundantly) doing the sampling. If sampling is cheap enough to do once, it should be cheap enough to do once per process/GPU in parallel.

That said, maybe there's a third option of sampling in one worker process and then broadcasting the result to each worker? It complicates the execution flow a bit, but it might be worth it.

[rjzamora]

I think it's okay to make this change in three phases:

1. Just do redundant sampling on each rank. We can use DGX & NVL72 benchmarks as a sanity check that this doesn't have a significant effect on perf.
2. We rip out most of the statistics infrastructure. In my opinion, that effort was a bit of a flop. I don't think we need all the lazy row-group sampling in the rapidsmpf-runtime world (lazy sampling was mostly used to detect high-cardinality groupby/distinct operations, but the dynamic-groupby/distinct is much more robust). The only sampling pieces we need are for accurate IO planning, and that can probably be done with simplified logic.
3. We use the simplifications in (2) to sample parquet metadata and row-groups in parallel.

Update: I opened #21843 to track the stats-infrastructure changes I think we should make.