Skip to content

[Feature][Metric] Custom SQL support export error data #560

@xxzuo

Description

@xxzuo

Search before asking

  • I had searched in the feature and found no similar feature requirement.

Description

The current custom_aggregate_sql metric requires users to write a complete aggregate SQL statement in the following format:

SELECT COUNT(1) AS actual_value FROM ${table} WHERE ...

This design has the following limitations:

  • Unable to output the actual error/invalid data records (isInvalidateItemsCanOutput()=false)
  • Users cannot intuitively identify which specific rows violate the data quality rules
  • Reduced efficiency in troubleshooting and data remediation

add a new metric type (e.g., custom_count_sql) that allows users to write only the SQL for querying error data:
SELECT * FROM ${table} WHERE <error_condition>

The system will automatically wrap it into an aggregate SQL during execution:

SELECT COUNT(*) AS actual_value FROM (
    SELECT * FROM ${table} WHERE <error_condition>
) dv_temp

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions