some confusion about using the get_metrics function

[zhangsunny]

Hi, I am developing my model with TSB-AD datasets.

According to the example (https://github.com/TheDatumOrg/TSB-AD/blob/ae42795ae23b0f332a716228ed5d218d7a4ce906/TSB_AD/main.py#L56), we can use the get_metrics function to calculate VUS-PR and VUS-ROC.

The get_metrics function has a parameter called slidingWindow (https://github.com/TheDatumOrg/TSB-AD/blob/ae42795ae23b0f332a716228ed5d218d7a4ce906/TSB_AD/evaluation/metrics.py#L3). Does this parameter represent the window size selected during model training?

In my experiments, I found that this window size has an impact on VUS-PR and VUS-ROC. Specifically, larger windows result in higher values for both metrics. Therefore, should we fix the slidingWindow parameter in get_metrics (e.g. the defualt 100) regardless of the window size chosen during model training to ensure a fair performance comparison?

[qhliu26]

Hi! The slidingWindow parameter in the get_metric function specifies the length of the buffer window used when calculating VUS. You can refer to this paper and its updated version here for more details.

By incorporating a buffer region around outlier boundaries, VUS accounts for ground-truth labeling tolerance and assigns higher anomaly scores to points near the boundary of an outlier segment. Therefore, the buffer window length is an important setting. In our experiments, we found that varying the buffer window length has a relatively small impact on the overall results.

If possible, could you share your results with us? We would love to take a closer look. Feel free to send them to us via email at liu.11085@osu.edu. Thank you!

[zhangsunny]

Thanks again for your patient and detailed responses. Your work has been extremely helpful for my current research, so I am now trying to delve deeper into the details. I was trying to send an email, but it was blocked beacuse of the attachments (attachments.zip).

Here are the experimental results I obtained using the file “172_SWaT_id_2_Sensor_tr_23700_1st_23800.csv” from TSB-AD-M:

​​Model​​: CNN (the typo metioned in self.mp = torch.nn.AdaptiveMaxPool1d(1) ( #31 ) has been corrected)

​​Hyperparameters​​: Recommended by HP_list.py {'window_size': 50, 'num_channel': [32, 32, 40]}

Observations​​: Increasing the slidingWindow parameter in “get_metrics” led to higher ​​VUS-PR​​, ​​VUS-ROC​​, and longer ​​evaluation time​​ as shown in the figure.

Attachments: The anomaly scores generated by the model and their corresponding ground truth labels are stored in output.pkl. The testing code that generated the above reuslts is in test_sth.py.

​​My question​​: Could there be any issues in my testing code (test_sth.py) that might affect these results? I would greatly appreciate your insights.

[jbirkesteen]

@qhliu26 Just to make sure; is slidingWindow=100 the buffer region used for evaluating results in your benchmark? I couldn't find this detailed in your paper, sorry if I missed it.

Thank you for your work with creating this valuable resource!