Support multiple S3 buckets (same path pattern) in a single raw table

[takahiro-oda-paypay]

Feature request

Use case: Ingest the same log schema from multiple S3 buckets (e.g. different regions or accounts) into a single raw (and downstream bronze) table, using the same path pattern per bucket.

Current behavior: Each CloudFiles load action has one path (one bucket + prefix). To combine multiple buckets, we have to:

• Define multiple load actions (one per bucket)
• Add a custom transform that unions the streams
• Use a custom template or inline actions

Desired behavior: Either of:

1. Multi-path load: Allow the path (or a new paths) field to accept multiple S3 URIs so a single load action reads from several buckets (e.g. path: ["s3://bucket-a/prefix/*", "s3://bucket-b/prefix/*"]), with the same schema and options applied to all.

2. Documented append-flow pattern: If multiple flowgroups writing to the same table (one create + N append flows) is already supported and stable, document it clearly so we can rely on “N flowgroups, same target table” instead of maintaining a custom multi-bucket template with manual union.

Impact: Would reduce custom templates and duplicated union logic for multi-bucket ingestion and make the “same schema, multiple buckets, one table” pattern a first-class option.

Thank you for considering this.

[Mmodarre]

Hi @takahiro-oda-paypay, thanks for raising this!

Great news — both approaches you outlined are already supported in Lakehouse Plumber. Let me walk you through them.

Option A: Multiple Append Flows (recommended for your use case)

LHP natively supports multiple write actions targeting the same streaming table. When you have multiple load actions writing to the same table, LHP generates a single dp.create_streaming_table() call with multiple @dp.append_flow() functions — one per source.

The key is the create_table flag. The first write action creates the table (create_table: true, which is the default), and subsequent writes append to it (create_table: false).

Example — two S3 buckets, one unified table:

pipeline: raw_ingestions
flowgroup: logs_multi_region

actions:
  # Load from US bucket
  - name: load_logs_us
    type: load
    readMode: stream
    source:
      type: cloudfiles
      path: "s3://my-bucket-us-east-1/logs/*.parquet"
      format: parquet
      options:
        cloudFiles.maxFilesPerTrigger: 100
    target: v_logs_us

  # Load from EU bucket
  - name: load_logs_eu
    type: load
    readMode: stream
    source:
      type: cloudfiles
      path: "s3://my-bucket-eu-west-1/logs/*.parquet"
      format: parquet
      options:
        cloudFiles.maxFilesPerTrigger: 100
    target: v_logs_eu

  # First write — creates the table
  - name: write_logs_us
    type: write
    source: v_logs_us
    write_target:
      type: streaming_table
      database: "{catalog}.{bronze_schema}"
      table: unified_logs
      create_table: true
    description: "Write US region logs to unified table"

  # Second write — appends to the same table
  - name: write_logs_eu
    type: write
    source: v_logs_eu
    write_target:
      type: streaming_table
      database: "{catalog}.{bronze_schema}"
      table: unified_logs
      create_table: false
    description: "Append EU region logs to unified table"

This generates:

dp.create_streaming_table(
    name="catalog.bronze.unified_logs",
    comment="Write US region logs to unified table"
)

@dp.append_flow(target="catalog.bronze.unified_logs", name="f_unified_logs_us")
def f_unified_logs_us():
    return spark.readStream.table("v_logs_us")

@dp.append_flow(target="catalog.bronze.unified_logs", name="f_unified_logs_eu")
def f_unified_logs_eu():
    return spark.readStream.table("v_logs_eu")

This is a production-ready, first-class pattern in LHP — documented in our Best Practices (BP-11.9 and BP-18.4) and Write Actions reference.

Option B: Templatize It

If you have many buckets following the same pattern, you can define a template for the load+write pair and invoke it multiple times — each targeting the same table. This eliminates boilerplate when scaling to N buckets.

Regarding multi-path loading (Option 1 in your issue)

We don't currently support array-valued path (or a paths field) on a single CloudFiles load action. The multiple append flows approach is the idiomatic way to achieve this in both Databricks and LHP, and it has the advantage of giving you independent checkpointing per source — so if one bucket has issues, the others continue processing normally.

That said, if there's a strong demand for a paths convenience shorthand (which would internally expand to multiple load + append flow pairs), we're open to considering it as a future enhancement. Let us know!