High Watermark Table

[Mmodarre]

New feature to create and update High watermark table

• This feature will help systems which use external tools (Azure Data Factory, etc.) for ingestion.
• LHP will create and maintain a table of high watermarks for tables which gets loaded by external integration tools so they can use the value as a high watermark for next ingestions

[federicoaguirre1]

Makes sense; currently, it is hard to maintain a watermark tracking value in Spark Declarative Pipelines. For example, if ingesting data from a SQLServer database table using an incremental approach, like SELECT * FROM TABLE WHERE watermark_column >= last_watermark_column_value