1.2 Release Planning

[vinothchandar]

Starting this discussion to plan out the 1.2 release. I propose moving to a faster release cycle this time, where we pick 1-2 key features per track and release when they are ready to go out.

Will start separate comments on each area and mention contributors I know, who are working on those.

[vinothchandar]

Spark

• MIT rewrite
• Explore removing caching on upsert path (JIRA)
• End-to-end row writing including Hudi streamer
• Spark 4.1 support (once Spark release is available)
• Deprecation and clean-up
• Remove glob support, unused relations

[vinothchandar]

datasource v2 - if someone can write a concise explanation of what we buy and how we retain the writer path for upsert as-is, it would be great. Else let's revisit later. ?

[vinothchandar]

nvm: just saw this. #13955 (comment) Will go over that.

[geserdugarov]

@vinothchandar , @yihua , about DSv2, I propose not to include this task for 1.2 release scope. I would say that milestone at 1.2 release would be POC implementation (draft PR) with benchmark results, and full design of corresponding RFC with writer path considering V1 and V2. And then, proper review could possibly take a lot of time, for instance, review of Spark 4 support took about 7 months (from Feb to Sep). So, the optimistic scenario is to prepare production ready functionality for 1.3 release. In negative scenarios, it will take more time depending on RFC design feedback.

[vinothchandar]

I agree. we can punt datasource v2 for 1.2.

[yihua]

Spark 4.1.0-preview4 is out (https://spark.apache.org/news/spark-4-1-0-preview4-released.html). We will likely need to add Spark 4.1 support for Hudi release 1.2.

[vinothchandar]

Storage Format

• Type system (RFC-99) (to have support for) discussion thread
• Blob type & Vector types for RFC-100
• New file format support to ensure for storing unstructured content (RFC-100) (Lance file format integration (board)
• Vector Index (ANN) (Review Suryas [PR]
• Spark TVF for Vector Search RFC 102; with End-End integration with RAG application to query tables using text

[vinothchandar]

@rahil-c @the-other-tim-brown @voonhous to confirm/add/remove

[the-other-tim-brown]

Do we want to include Variant support or is that expanding the scope too far?

[vinothchandar]

do we know the scope. but yes aligning with how the type is defined in RFC-99 would be ideal.

[vinothchandar]

Storage Engine

• Performance certification of SI/RLI reads/writes.
• Schema abstraction – don’t use Avro directly.
• HoodieStorage enhancement on reader and writer paths

[yihua]

There are HoodieStorage enhancements that @mansipp and @CTTY would like to contribute.

[vinothchandar]

Flink

• Flink CDC Hudi sink connector: Implement Flink CDC connector #17649
• Virtual column read support, e.g, the _hoodie_commit_time: Support reading virtual metadata columns for flink #14308
• Flink 2.1 support Support Flink 2.1 #17533
• Async clustering for Flink: Flink async clustering / compaction gaps #17639
• RLI/SI support for Flink reading/writing: Support record index for Flink writer #17647; docs: RFC-102 - RLI support for Flink streaming #17610
• New Flink source with FLIP 27 [Umbrella] Hudi Flink Unified Source Implementation #17038
• LSM file layout for streaming: New LSM file layout #14310; docs: RFC-103 Hudi LSM Tree Layout #17827

[vinothchandar]

@danny0405 what about async clustering can we write out an issue that describes current state /gaps?

[danny0405]

@vinothchandar it is already supported, I reached out to Uber/Peter and he said something related to the notification between clustering schedule and execution, that is beyond the scope of Flink though, needs further clarification.

[vinothchandar]

@HuangZhenQiu we would like to get the 1.2 Flink items finalized. Can you please chime in here?

[vinothchandar]

• Dynamic bucket scaling for bucket index: (still planning)

Pushed to after 1.2.

[vinothchandar]

@shangxinli Besides RLI/upsert support, what do you see as biggest needs

[parisni]

May contributor suggest stuff ? If so let me propose full support for table documentation (doc, flat and nested comments) for both reading hudi and writing into hudi from spark, trino or any hms/glue/s3 hudi compliant readers

• fix: aws glue sync set comment #14234
• [HUDI-5533] Support spark nested columns comments #14235

Comment support has been commited back in 2022 #4960 but the feature has never been working due to lack of suppprt for comment extraction from the hudi avro schema.

The nested comment suppports allows spark io to propagate the comments transparently thought the transformation lineage, which is AFAIK unique among acid format (delta/iceberg) do not support nested comment, only first level.

If this is of interest i can also work to backport this work for 0.15 since it can be considered as a bug fix.

[vinothchandar]

yes absolutely @parisni ..

[danny0405]

yes, it's useful, I can help for the code review of these two PRs.

[vinothchandar]

We can give this 2 days, till EOD Friday PST. and freeze 1.2 scope as described here so far.

[vinothchandar]

I think we can close this discuss thread. with the agreed upon scope across each area.