feat: add sink.parallelism support for OceanBase JDBC connector#138
Open
yuanoOo wants to merge 5 commits intooceanbase:mainfrom
Open
feat: add sink.parallelism support for OceanBase JDBC connector#138yuanoOo wants to merge 5 commits intooceanbase:mainfrom
yuanoOo wants to merge 5 commits intooceanbase:mainfrom
Conversation
Add support for configuring sink parallelism via 'sink.parallelism' option. This allows users to set a custom parallelism for the sink operator in SQL DDL or Table Hints. Changes: - Add SINK_PARALLELISM config option in ConnectorOptions - Modify SinkProvider to support parallelism via getParallelism() method - Register the option in OceanBaseDynamicTableSinkFactory - Pass parallelism parameter in OceanBaseDynamicTableSink - Add unit test for SinkProvider parallelism - Add integration test for sink.parallelism configuration
- Fix compilation error: pass parallelism to SinkProvider in OBKVHBaseDynamicTableSink - Register SINK_PARALLELISM in OBKVHBaseDynamicTableSinkFactory - Add validation that sink.parallelism must be positive - Add unit test for invalid parallelism values - Clean up integration test comments
Add verification of sink parallelism in integration test by checking the JSON execution plan contains the expected parallelism value.
Print execution plan content to help diagnose parallelism verification failure. Also add more flexible assertion patterns.
The parallelism was correctly parsed but not applied to the DataStreamSink. Added setParallelism() call in consumeDataStream() to properly set the sink parallelism.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
sink.parallelismoption in SQL DDL or Table HintsChanges
SINK_PARALLELISMconfig option inConnectorOptionsSinkProviderto support parallelism viagetParallelism()methodOceanBaseDynamicTableSinkFactoryOceanBaseDynamicTableSinkSinkProviderparallelismsink.parallelismconfigurationUsage
DDL Configuration
Table Hints
Test Plan
AbstractDynamicTableSinkTestverifiesgetParallelism()returns correct valuetestSinkParallelisminOceanBaseMySQLConnectorITCase