Skip to content

[AURON #2229] Silent fallback to empty partitions may cause data loss#2230

Open
guixiaowen wants to merge 2 commits intoapache:masterfrom
guixiaowen:foriceberg_2229
Open

[AURON #2229] Silent fallback to empty partitions may cause data loss#2230
guixiaowen wants to merge 2 commits intoapache:masterfrom
guixiaowen:foriceberg_2229

Conversation

@guixiaowen
Copy link
Copy Markdown
Contributor

#2229

Which issue does this PR close?

Closes #2229

Rationale for this change

The current implementation silently falls back to Seq.empty when both DataSource V2 and reflective partition retrieval fail. This may lead to empty input partitions without any failure signal, potentially causing silent data loss or empty query results.

Problem Description

In inputPartitions(exec: BatchScanExec), when partition planning fails:

V2 API failure → caught and ignored
Reflection failure → caught and ignored
Final fallback → Seq.empty

catch { case t: Throwable => logWarning(...) Seq.empty }

and

catch { case t: Throwable => logWarning(...) Seq.empty }

Risk

Returning an empty partition list is dangerous because:

  1. Silent data loss
    Downstream execution may interpret empty partitions as:

  2. no data available
    valid empty scan result

→ leads to incorrect empty query results without failure

Hard to detect in production
Only warning logs are emitted
No exception or job failure
Easy to miss in large pipelines

  1. Misleading semantics
    A scan failure (cannot plan partitions) is semantically different from:

dataset is actually empty

But both are currently indistinguishable.

What changes are included in this PR?

Expected Behavior

When both:

V2 planInputPartitions
reflection-based partition retrieval

fail, the system should:

Fail fast with a clear exception:

throw new IllegalStateException( s"Failed to plan input partitions for ${exec.getClass.getName}")

Are there any user-facing changes?

Nothing.

How was this patch tested?

Depends on existing unit tests.

@guixiaowen guixiaowen changed the title [AURON #2229] Silent fallback to empty partitions may cause data loss… [AURON #2229] Silent fallback to empty partitions may cause data loss May 3, 2026
s"Failed to obtain input partitions via reflection for ${exec.getClass.getName}.",
t)
Seq.empty
throw new IllegalStateException(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are throwing an exception anyways, do we need to catch it in L189?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Silent fallback to empty partitions may cause data loss

2 participants