Support for Multiple Columns in cluster_by in DLT Meta Framework

[shishupalgeek]

While working with the DLT Meta framework, I noticed that the Bronze And Silver table cluster_by parameter currently allows only a single column to be specified as a clustering key.

According to the official Databricks Create Streaming Table documentation, cluster_by supports defining a list of columns to enable liquid clustering on a table. This allows tables to be organized more efficiently using multiple clustering keys.

I recommend adding support for multiple columns in cluster_by in the DLT Meta framework. This enhancement would improve table optimization and make the framework consistent with standard Spark Declarative Pipeline capabilities.

Reference:
Databricks Spark Declarative Pipeline documentation:
https://learn.microsoft.com/en-in/azure/databricks/ldp/developer/ldp-python-ref-streaming-table

Excerpt from documentation:

cluster_by | list | Enable liquid clustering on the table and define the columns to use as clustering keys. See Use liquid clustering for tables.

[ravi-databricks]

Hi @shishupalgeek,

I have verified that this feature is working correctly in DLT-META. Please follow the steps below to reproduce:

1. Run the Append Flow Autoloader file metadata demo.

   • Above demo uses this onboarding file.

2. Check Configuration: Ensure you are using the correct keys in your onboarding file:

   • Bronze: "bronze_cluster_by"
   • Silver: "silver_cluster_by"
   • Example: "bronze_cluster_by": ["id", "email"]
   • (You can check the full documentation here).

3. Verify Results: Once the demo finishes, run the following command to confirm the clustering was applied:
   DESCRIBE DETAIL <your_bronze_table_name>

   • You should see the clusterBy values populated as shown in clusteringColumns as shown in the screenshot below:

[shishupalgeek]

thanks @ravi-databricks , i will test it again.