Skip to content

[Feature] concurrent write of struct sinker #433

@caiq1nyu

Description

@caiq1nyu

Motivations

the current structure's write model only supports serial writes(serial_parallelizer.rs), which can lead to performance bottlenecks when there are many tables.

[extractor]
extract_type=struct

[sinker]
sink_type=struct

[parallelizer]
parallel_type=serial

direct and rough concurrency may cause data problems, such as creating an index first but the table does not exist

Solution

  • provides a new parallelizer implementation that includes data conflict management in concurrent scenarios.
  • mark conflict information in advance in the data fetched in the extractor (xx-struct-extractor.rs)

'tables' depend on 'database';
'indexs' depend on 'tables';
'tables have FK' deplend on 'other tables' and so on

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions