Confused by behavior of toMatchScreenshot

[kylemh]

Hey folks! Congrats on v4. I've adopted and was trying out toMatchScreenshot to make some visual regression diff tests and the output was surprising to me...

so, there's a .vitest-attachments folder with
.vitest-attachments/Reference-screenshot-87cb0abd024e566dc796aabea651788d6c094057.png

then there's 2 co-located screenshots alongside the relevant test file path:
src/components/_ui/__tests__/__screenshots__/RadioGroup.browser.test.tsx/radio-group-chromium-darwin.png
src/components/_ui/__tests__/__screenshots__/RadioGroup.browser.test.tsx/RadioGroup-should-have-no-visual-regressions-1.png

this is all for just one test:

import { render } from '@testing-library/react';
import { describe, expect, it } from 'vitest';
import { page } from 'vitest/browser';

import { RadioGroup } from '@/components/_ui/RadioGroup';
import { RADIO_GROUP } from '@/testIDs';

describe('RadioGroup', () => {
  it('should have no visual regressions', async () => {
    render(
      <RadioGroup
        options={[
          { value: 'option1', label: 'Option 1' },
          { value: 'option2', label: 'Option 2' },
          { value: 'option3', label: 'Option 3' },
        ]}
      />,
    );

    // capture and compare screenshot
    await expect(page.getByTestId(RADIO_GROUP)).toMatchScreenshot('radio-group');
  });
});

why do all of these exist? which do I commit for CI?

[kylemh]

My config, if it matters:

/// <reference types="vitest/config" />

import { defineConfig } from 'vitest/config';
import react from '@vitejs/plugin-react';
import { playwright } from '@vitest/browser-playwright';
import tailwindcss from '@tailwindcss/vite';

import tsconfigPaths from 'vite-tsconfig-paths';

/** @see https://vitejs.dev/config/ */
export default defineConfig({
  plugins: [react(), tailwindcss(), tsconfigPaths()],
  test: {
    globals: true,
    projects: [
      {
        extends: true, // also use root config
        test: {
          include: ['src/**/*.test.{ts,tsx}'], // if this changes, please update ESLint config too
          exclude: ['src/**/*.browser.test.{ts,tsx}'],
          environment: 'jsdom',
          setupFiles: ['vitest.browser.setup.tsx', 'vitest.jsdom.setup.ts'],
        },
      },
      {
        extends: true, // also use root config
        test: {
          include: ['src/**/*.browser.test.{ts,tsx}'], // if this changes, please update ESLint config too
          setupFiles: ['vitest.browser.setup.tsx'],
          browser: {
            enabled: true,
            provider: playwright(),
            instances: [{ browser: 'chromium' }],
            headless: true,
            // screenshotFailures: false,
          },
        },
      },
    ],
    coverage: {
      include: ['src/**/*.{ts,tsx}'],
      provider: 'istanbul',
      reporter: ['text', 'lcov'],
    },
  },
});

[finico]

@kylemh Likely it's browser.screenshotFailures behavior
https://vitest.dev/config/browser/screenshotfailures.html

[finico]

so..

src/components/_ui/__tests__/__screenshots__/RadioGroup.browser.test.tsx/radio-group-chromium-darwin.png is a reference file

.vitest-attachments/Reference-screenshot-87cb0abd024e566dc796aabea651788d6c094057.png is a actual file

src/components/_ui/__tests__/__screenshots__/RadioGroup.browser.test.tsx/RadioGroup-should-have-no-visual-regressions-1.png is a screenshot of failure for the test

which do I commit for CI?

only reference files

[kylemh]

Thanks for answering the question. I guess I'll just have to match for .gitignore

but I don't think both a reference file and a failure file should exist since every rich source control management viewer can highlight the differences of a change in an image file.

#9236 (comment)

[finico]

It may depend on the testing scenario. If a test checks a specific element for visual regression using a locator, then a screenshot of the entire viewport is taken when an error occurs to understand the context. Therefore, I separated the tests into projects, similar to your config above, but added a visual (included by pattern src/**/*.visual.test.{ts,tsx}) and disabled screenshotFailures in it.

[kylemh]

That sounds like it'll work for me, too. Thanks!

[macarie]

As said in #9236, failure screenshots won't be captured anymore in v4.1 if toMatchScreenshot produced any screenshots.

.vitest-attachments is the folder used to store side attachments and may be ignored and optionally uploaded as an artifact from CI to have access to actuals and diffs.

Creating a different (Vitest) project dedicated to visual regression testing with a different set of options is good and might improve the stability of the screenshots, especially if it's set to run in headless mode.