Concerns about label quality issues in TSB-AD-U

[richard-tang199]

Description:
Thank you for open-sourcing the dataset. However, during data visualization and analysis, I identified several quality issues that need attention:

Inaccurate Annotations
Datasets 001-005: Incorrect anomaly range annotations

Abnormally High Anomaly Density
Datasets 149-168: Anomaly density exceeds practical observation patterns

Uninterpretable Anomalies
Datasets 009 & 016: Difficult to interpret the underlying reasons of anomalies

Trivial Anomalies
Datasets 029-139: Anomaly patterns are large or small values

Improper Dataset Split
Dataset 145: Unreasonable split between training and testing sets

[qhliu26]

Hello Richard,

Thank you so much for reviewing the labeling issues! We’d like to share our thought process below and welcome any feedback or discussion. Please also let us know if there are any datasets you'd recommend for inclusion in TSB-AD.

• Issues regarding inaccurate annotations: Most issues arise from anomaly regions being marked larger than expected, which is primarily due to differing labeling practices. For example, in the NAB dataset, the original annotations label entire regions as anomalies. Since re-labeling these annotations is beyond our scope—and would be somewhat arbitrary—we choose to retain the original annotations as provided.

• Issues regarding abnormally high anomaly density: We noticed this as well! Since many of the anomalies are individual time steps, the overall anomaly ratio remains relatively low (below 10%). While many of the point anomalies are fairly simple, we retain them in the collection to ensure comprehensive coverage of representative cases.

• Issues regarding uninterpretable anomalies: We acknowledge that, upon visual inspection, these anomaly patterns are subtle and may be difficult for humans to perceive. However, according to our filtering criteria, at least one anomaly detector was able to identify them, indicating that these anomalies align with the heuristics used by certain detectors.

• Issues regarding trivial anomalies: As shown in the attached figure, some anomalies are indeed relatively trivial. However, we include them to ensure comprehensive coverage across datasets. We also expect that a strong anomaly detector should not only handle challenging cases but also reliably detect these simpler anomalies.

• Issues regarding dataset splitting: the definitions of unsupervised and semi-supervised approaches #6 (comment)

[richard-tang199]

Hello qhliu26,
Thank you for your detailed response and for addressing the concerns regarding the label quality issues in TSB-AD-U. We appreciate your insights and the effort you've put into explaining the rationale behind the labeling decisions.