Addon A11y: Create a11y test provider and revamp a11y addon

[valentinpalkovic]

Closes #

What I did

Summary

This PR introduces significant updates to the @storybook/addon-a11y for improved integration with Storybook's component testing feature, powered by Vitest. The enhancements streamline accessibility testing, provide better developer tooling, and address long-standing issues. Key changes include configuration updates, parameter deprecations, and integration with global reporting features.

Changelog

New Features

1. Integration with Component Testing: Introduced seamless accessibility checks within component tests.
   Added a11y addon annotations to .storybook/vitest.setup.ts for automatic setup during upgrades.
2. Violation Severity Levels: Added the ability to categorize violations as warnings or errors based on their impact levels.
3. Enhanced UI and Status Discrepancy Messages: Added a new TestDiscrepancyMessage component to highlight CLI-browser testing discrepancies in the UI.
   Improved sidebar accessibility status reporting.
4. Global Parameters for a11y: Introduced support for globals.a11y.manual to globally control manual execution of accessibility tests.
5. Reporter API: Introduced a generic reporting API (ReporterAPI) to handle reports in a structured manner across multiple contexts. Reports now include attributes such as id, status (e.g., failed, passed), and result data. Stories leverage the reporting mechanism to aggregate results, ensuring detailed tracking of failures or warnings.
6. New STORY_FINISHED event: Introduced a STORY_FINISHED event that consolidates the end state of a story, emitting success or failure with associated reports.
7. afterEach: Added an applyAfterEach hook to StoryStore and StoryRender to facilitate post-test actions (e.g., cleanup, validation). Additionally, modified the test lifecycle to account for afterEach actions in both stories and meta-level annotations.

Deprecations

1. Deprecated parameters.a11y.manual in favor of globals.a11y.manual.

Bug Fixes

• Fixed false positives in region axe rule during component-level testing by disabling it by default.

Migration Notes

For Projects Updating to v8.5:

• Automatic upgrade via npx storybook@latest upgrade integrates the a11y addon into component tests.
• If upgrading manually, update .storybook/vitest.setup.ts:

+import * as a11yAddonAnnotations from '@storybook/addon-a11y/preview';
const annotations = setProjectAnnotations([
+ a11yAddonAnnotations,
  previewAnnotations,
]);
beforeAll(annotations.beforeAll);

• Update your parameters.a11y.manual usage to globals.a11y.manual.

Configuring Violation Impact Levels:

Set the warnings parameter for specific stories:

export const parameters = {
  a11y: {
    warnings: ['minor', 'moderate'],
  },
};

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

• stories
• unit tests
• integration tests
• end-to-end tests

Manual testing

Setup

Case 1: Setup in projects where @experimental-addon-test and @addon-a11y are already installed

1. npx storybook@<canary> upgrade
2. An automigration should be triggered to add the a11y annotations to .storybook/vitest.setup.ts

Case 2: Setup in projects where @experimental-addon-test and @addon-a11y are already installed, but with undetected
.storybook/vitest.setup.ts file

1. Remove or move .storybook/vitest.setup.ts
2. npx storybook@<canary> upgrade
3. An automigration should be triggered telling the user to make manual adjustments in vitest.setup.ts, since we couldn't detect it automatically

Case 3: Add @storybook/experimental-addon-test in a project where addon-a11y is already set up (Can only be tested in an alpha release, because the add command in canaries does not work):

1. Remove @storybook/experimental-addon-test if available and all correlated files (.storybook/vitest.setup.ts, vitest.workspace,...)
2. npx storybook@<canary> upgrade
3. npx storybook add @storybook/addon-a11y (if not already available)
4. npx storybook add @storybook/experimental-addon-test -> It should set up .storybook/vitest.setup.ts with addon-a11y annotations.

Features

Prerequisites

1. Make sure the addon is either automatically set up or make the necessary changes in .storybook/vitest.setup.ts if the auto migrations or postinstall scripts didn't work.

+import * as a11yAddonAnnotations from '@storybook/addon-a11y/preview';
const annotations = setProjectAnnotations([
+ a11yAddonAnnotations,
  previewAnnotations,
]);
beforeAll(annotations.beforeAll);

Storybook: Accessibility violations should not be reported per default

1. Start Storybook
2. Make sure the Accessibility checkbox is not ticked in the Testing Module
   
3. Run tests -> Accessibility errors should not be reported in the sidebar
4. Go to a component, open the Accessibility tab, and errors should be reported in the tab if any
   

Storybook: Accessibility violations should be reported as errors

1. Start Storybook
2. Make sure the Accessibility checkbox is ticked in the Testing Module
3. Run tests -> Accessibility errors should be reported as errors in the sidebar
   
4. Go to story with violations and click on Accessibility tests in the story context menu to get redirected to the Story.
   
5. The a11y addon tab should automatically open.

Tags support - Disable accessibility runs

1. Go to your .storybook/preview.js file and set the !a11ytest tag

// .storybook/preview.js
export const tags = ['!a11ytest'];

2. Start Storybook (Restart)
3. Make sure the Accessibility checkbox is ticked in the Testing Module
4. Run tests -> Accessibility violations shouldn't be reported
5. The a11y panel should still work, though

Tags support - Partially re-enable accessiblity runs

1. Go to your .storybook/preview.js file and set the !a11ytest tag

// .storybook/preview.js
export const tags = ['!a11ytest'];

2. Go to a story which has accessibility violations and add the a11ytest tag

export const Button = {
  tags: ['a11ytest']
}

Test discrepancy

1. Enable the region rule, which is deactivated per default -> This will lead to overreporting in a portable stories environment

// .storybook/preview.ts
export const parameters = {
  a11y: {
    config: {
      rules: [
        {
          id: 'region',
          enabled: true,
        },
      ],
    },
  },
};

2. Start Storybook
3. Make sure the Accessibility checkbox is ticked in the Testing Module
4. Run tests -> Much more accessibility issues should be now reported.
5. Go to a story, which didn't have previously a11y violations and open the a11y panel. It should have a discrepancy message, pointing to the docs (docs link might not work yet, since the docs aren't published yet)
   

Documentation

• Add or update documentation reflecting your changes
• If you are deprecating/removing a feature, make sure to update
  MIGRATION.MD

Checklist for Maintainers

• When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli-storybook/src/sandbox-templates.ts

• Make sure this PR contains one of the labels below:

  Available labels

  • bug: Internal changes that fixes incorrect behavior.
  • maintenance: User-facing maintenance tasks.
  • dependencies: Upgrading (sometimes downgrading) dependencies.
  • build: Internal-facing build tooling & test updates. Will not show up in release changelog.
  • cleanup: Minor cleanup style change. Will not show up in release changelog.
  • documentation: Documentation only changes. Will not show up in release changelog.
  • feature request: Introducing a new feature.
  • BREAKING CHANGE: Changes that break compatibility in some way with current major version.
  • other: Changes that don't fit in the above categories.

🦋 Canary release

This pull request has been released as version 0.0.0-pr-29643-sha-dd3f1df2. Try it out in a new sandbox by running npx storybook@0.0.0-pr-29643-sha-dd3f1df2 sandbox or in an existing project with npx storybook@0.0.0-pr-29643-sha-dd3f1df2 upgrade.

More information

Published version	0.0.0-pr-29643-sha-dd3f1df2
Triggered by	@kasperpeulen
Repository	storybookjs/storybook
Branch	valentin/unified-a11y-testing
Commit	dd3f1df2
Datetime	Fri Dec 6 16:36:04 UTC 2024 (1733502964)
Workflow run	12202532250

To request a new release of this pull request, mention the @storybookjs/core team.

core team members can create a new canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=29643

name	before	after	diff	z	%
createSize	0 B	0 B	0 B	-	-
generateSize	77.7 MB	77.7 MB	0 B	-0.82	0%
initSize	130 MB	133 MB	2.38 MB	19.67	1.8%
diffSize	52.7 MB	55.1 MB	2.38 MB	18.09	4.3%
buildSize	6.87 MB	6.87 MB	4.33 kB	1.44	0.1%
buildSbAddonsSize	1.51 MB	1.51 MB	2.22 kB	Infinity	0.1%
buildSbCommonSize	195 kB	195 kB	0 B	-	0%
buildSbManagerSize	1.86 MB	1.86 MB	434 B	10.01	0%
buildSbPreviewSize	0 B	0 B	0 B	-	-
buildStaticSize	0 B	0 B	0 B	-	-
buildPrebuildSize	3.57 MB	3.57 MB	2.66 kB	56.48	0.1%
buildPreviewSize	3.3 MB	3.3 MB	1.67 kB	1.39	0.1%
testBuildSize	0 B	0 B	0 B	-	-
testBuildSbAddonsSize	0 B	0 B	0 B	-	-
testBuildSbCommonSize	0 B	0 B	0 B	-	-
testBuildSbManagerSize	0 B	0 B	0 B	-	-
testBuildSbPreviewSize	0 B	0 B	0 B	-	-
testBuildStaticSize	0 B	0 B	0 B	-	-
testBuildPrebuildSize	0 B	0 B	0 B	-	-
testBuildPreviewSize	0 B	0 B	0 B	-	-

name	before	after	diff	z	%
createTime	7.1s	24.8s	17.7s	1.93	🔺71.3%
generateTime	20s	20.6s	666ms	0.14	3.2%
initTime	13.9s	13.2s	-649ms	-0.26	-4.9%
buildTime	8.6s	9.6s	936ms	1.36	🔺9.7%
testBuildTime	0ms	0ms	0ms	-	-
devPreviewResponsive	5.7s	5.1s	-631ms	0.13	-12.2%
devManagerResponsive	4.2s	3.6s	-620ms	-0.15	-17.1%
devManagerHeaderVisible	634ms	582ms	-52ms	0.31	-8.9%
devManagerIndexVisible	701ms	658ms	-43ms	0.18	-6.5%
devStoryVisibleUncached	2s	1.8s	-171ms	0.45	-9.1%
devStoryVisible	702ms	657ms	-45ms	0.41	-6.8%
devAutodocsVisible	700ms	477ms	-223ms	-0.66	-46.8%
devMDXVisible	588ms	505ms	-83ms	0.02	-16.4%
buildManagerHeaderVisible	699ms	551ms	-148ms	-0.2	-26.9%
buildManagerIndexVisible	822ms	652ms	-170ms	0.05	-26.1%
buildStoryVisible	629ms	512ms	-117ms	-0.04	-22.9%
buildAutodocsVisible	527ms	435ms	-92ms	-0.23	-21.1%
buildMDXVisible	532ms	414ms	-118ms	-0.41	-28.5%

Greptile Summary

Based on the extensive changes in this pull request, here's a concise summary:

Major revamp of Storybook's accessibility (a11y) addon with improved test integration and reporting capabilities.

• Added new test lifecycle hooks with experimental_afterEach for post-test validation in code/core/src/preview-api/modules/store
• Introduced ReporterAPI in code/core/src/preview-api/modules/store/reporter-api.ts for standardized test result reporting
• Added test discrepancy detection in code/addons/a11y/src/components/TestDiscrepancyMessage.tsx to highlight differences between CLI and browser test results
• Deprecated parameters.a11y.manual in favor of globals.a11y.manual for runtime configuration
• Added support for categorizing violations as warnings vs errors based on impact levels in code/addons/a11y/src/preview.tsx

The changes look well-structured but should be carefully reviewed for error handling and race conditions in the test lifecycle management.

_{💡 (5/5) You can turn off certain types of comments like style here!}

[nx-cloud]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit b20b03b. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

---

🟥 Failed Commands
nx run-many -t build --parallel=3

---

Sent with 💌 from NxCloud.

[kasperpeulen]

You don't need to set the tag if you already disabled the test with the global.