Performance degradation caused by normalization

[BoZenKhaa]

Performance degradation on PCA

I was getting unexpectedly poor performance from PCA on the Exathlon data (using the VUS-PR metric):

	TSB-AD - latest commit	Values reported in the paper	Difference
TranAD	0.95	0.10	0.86
OFA	0.85	0.58	0.28
CNN	0.95	0.68	0.27
LSTMAD	0.96	0.82	0.14
OmniAnomaly	0.97	0.84	0.13
USAD	0.97	0.84	0.13
RobustPCA	0.81	0.77	0.04
AnomalyTransformer	0.14	0.10	0.04
AutoEncoder	0.91	0.91	0.00
IForest	0.32	0.35	-0.04
PCA	0.53	0.95	-0.42

First column are values I obtained with the latest commit of TSB-AD, second are values from the publication.

The improvement in most methods can probably be ascribed to the fixes (e.g. use of correct hyperparameters) since publication. However, there might be a bug affecting PCA and possibly other methods as well:

Bug (in PCA?)

At least part of the issue might be normalization introduced in a79f315:

        # models/PCA.py
        X = Window(window = self.slidingWindow).convert(X)
        if self.normalize: 
            if n_features == 1:
                X = zscore(X, axis=0, ddof=0)
            else: 
                X = zscore(X, axis=1, ddof=1) #<--- 2nd issue
                
        # validate inputs X and y (optional)
        X = check_array(X)
        self._set_n_classes(y)

        # PCA is recommended to use on the standardized data (zero mean and
        # unit variance).
        if self.standardization: #<--- 1st issue
            X, self.scaler_ = standardizer(X, keep_scalar=True)

In the example of PCA above:

1. Normalization is applied twice, once for the normalization flag and once for the standardization flag.
2. X = zscore(X, axis=1, ddof=1) seems to apply normalization independently on each window. Each window contains multiple features and has form [feat0_t0, feat0_t1, ..., feat0_tw, feat1_t0, feat1_t1, ..., feat1_tw, ...].

It is not intuitive for me what No 2. is trying to achieve, but it does not seem correct PCA. Independent application of the z-score to each time window e.g. messes up information on absolute magnitude of features between time windows, and overall seems to only remove information from the data.

Also, I think normalization might not be needed for IsolationForest forest at all? For other methods, I am currently unsure I can't tell for sure without looking into them more.

Fix

Two parts to this:

1. In my opinion, PCA should not have both "normalization" and "standardization". I think removing "normalization" code and renaming "standardization" to "normalization" would be appropriate to be consistent with other methods. Even though I think "standardization" is actually a better term here.
2. I think X = zscore(X, axis=1, ddof=1) is a bug at least for some methods (PCA, IForest). Naively, I believe changes from a79f315 could be replaced to use StandardScaler in standardizer along the columns for all the methods involved.

I can prepare a PR for either 1. or 1. and 2., if that is welcome.

PS: thank you for an awesome project of such a large scope <3

[BoZenKhaa]

I ran the latest commit PCA with and without the "normalization" code on the Exathlon dataset, the result is avereage VUS-PR:

• Without normalization: 0.9708
• With normalization: 0.5408

[qhliu26]

Thank you for the careful investigation and for sharing your insights. The motivation for applying z-score normalization to each subsequence (i.e., sliding window) originates from distance-based approaches such as Matrix Profile, where z-score normalization is used by default as part of the preprocessing pipeline. To ensure a fair comparison, we therefore applied z-score normalization to the remaining methods as well, unless a method explicitly specifies a different normalization strategy. In practice, we also observed that normalization plays a significant role: it can substantially alter the shape of the signal, which in some cases is beneficial, but in others may backfire.

Regarding the multivariate processing pipeline, the current approach - concatenating windows across multiple dimensions into a single temporal dimension - is a temporary solution to adapt methods such as PCA to multivariate time series. We agree that, in this setting, normalization can indeed backfire by distorting the underlying information, and that omitting normalization may lead to better performance. An alternative solution is to treat each time point across all channels as a single data instance, which corresponds to the processing pipeline we previously used for PCA. Hope this can be helpful for you!

[BoZenKhaa]

Thank you very quick response and for adding this context! Now I see where this is coming from. With a univariate method like Matrix Profile, this makes perfect sense to me.

However, I am still unsure for which method it would be useful in the multivariate case, even for subsequence of length 1 - which is how I understand your alternate solution.

And specifically with current PCA defaults, applying z-score twice seems off. First, there is z-score on subsequences which I can see as reasonable for univariate datasets. But then there is the z-score on each column of the matrix of sub-sequences with the standardizer (which as far as I can tell is a StandardScaler from scikit that does z-score normalization on columns). Seems to me that should not happen in either mutlivariate or univariate case. At the very least, there should be different parameter default for PCA (normalization=true, standardization=false), or the defaults should be set based on whether the dataset is univariate or multivariate.

Finally, if I have the opportunity to pick your brain, I have two questions:

1. Across the board, for mutlivariate experiments, do you think there is a method that could benefit from the current implementation of the z-score normalization applied to each subseries concatenated across channels?
2. In case of mutlivariate PCA, in my mutlivariate usecase, I am thinking the reasonable thing would be to use the StandardScaler z-score (on features) before applying the time windows and concatenation. Do you see some issue with that?

Many thanks for taking the time to respond!