A11y: Create a11y test provider and revamp a11y addon

MIGRATION.md

@@ -1,5 +1,8 @@
 <h1>Migration</h1>
 
+- [From version 8.4.x to 8.5.x](#from-version-84x-to-85x)
+  - [Addon-a11y: Integration into `component tests`](#addon-a11y-integration-into-component-tests)
+  - [Addon-a11y: Deprecated `parameters.a11y.manual`](#addon-a11y-deprecated-parametersa11ymanual)
 - [From version 8.2.x to 8.3.x](#from-version-82x-to-83x)
   - [Removed `experimental_SIDEBAR_BOTTOM` and deprecated `experimental_SIDEBAR_TOP` addon types](#removed-experimental_sidebar_bottom-and-deprecated-experimental_sidebar_top-addon-types)
   - [New parameters format for addon backgrounds](#new-parameters-format-for-addon-backgrounds)
@@ -105,17 +108,17 @@
     - [Tab addons cannot manually route, Tool addons can filter their visibility via tabId](#tab-addons-cannot-manually-route-tool-addons-can-filter-their-visibility-via-tabid)
     - [Removed `config` preset](#removed-config-preset-1)
 - [From version 7.5.0 to 7.6.0](#from-version-750-to-760)
-  - [CommonJS with Vite is deprecated](#commonjs-with-vite-is-deprecated)
-  - [Using implicit actions during rendering is deprecated](#using-implicit-actions-during-rendering-is-deprecated)
-  - [typescript.skipBabel deprecated](#typescriptskipbabel-deprecated)
-  - [Primary doc block accepts of prop](#primary-doc-block-accepts-of-prop)
-  - [Addons no longer need a peer dependency on React](#addons-no-longer-need-a-peer-dependency-on-react)
+    - [CommonJS with Vite is deprecated](#commonjs-with-vite-is-deprecated)
+    - [Using implicit actions during rendering is deprecated](#using-implicit-actions-during-rendering-is-deprecated)
+    - [typescript.skipBabel deprecated](#typescriptskipbabel-deprecated)
+    - [Primary doc block accepts of prop](#primary-doc-block-accepts-of-prop)
+    - [Addons no longer need a peer dependency on React](#addons-no-longer-need-a-peer-dependency-on-react)
 - [From version 7.4.0 to 7.5.0](#from-version-740-to-750)
-  - [`storyStoreV6` and `storiesOf` is deprecated](#storystorev6-and-storiesof-is-deprecated)
-  - [`storyIndexers` is replaced with `experimental_indexers`](#storyindexers-is-replaced-with-experimental_indexers)
+    - [`storyStoreV6` and `storiesOf` is deprecated](#storystorev6-and-storiesof-is-deprecated)
+    - [`storyIndexers` is replaced with `experimental_indexers`](#storyindexers-is-replaced-with-experimental_indexers)
 - [From version 7.0.0 to 7.2.0](#from-version-700-to-720)
-  - [Addon API is more type-strict](#addon-api-is-more-type-strict)
-  - [Addon-controls hideNoControlsWarning parameter is deprecated](#addon-controls-hidenocontrolswarning-parameter-is-deprecated)
+    - [Addon API is more type-strict](#addon-api-is-more-type-strict)
+    - [Addon-controls hideNoControlsWarning parameter is deprecated](#addon-controls-hidenocontrolswarning-parameter-is-deprecated)
 - [From version 6.5.x to 7.0.0](#from-version-65x-to-700)
   - [7.0 breaking changes](#70-breaking-changes)
     - [Dropped support for Node 15 and below](#dropped-support-for-node-15-and-below)
@@ -141,7 +144,7 @@
     - [Deploying build artifacts](#deploying-build-artifacts)
       - [Dropped support for file URLs](#dropped-support-for-file-urls)
       - [Serving with nginx](#serving-with-nginx)
-      - [Ignore story files from node_modules](#ignore-story-files-from-node_modules)
+      - [Ignore story files from node\_modules](#ignore-story-files-from-node_modules)
   - [7.0 Core changes](#70-core-changes)
     - [7.0 feature flags removed](#70-feature-flags-removed)
     - [Story context is prepared before for supporting fine grained updates](#story-context-is-prepared-before-for-supporting-fine-grained-updates)
@@ -155,7 +158,7 @@
     - [Addon-interactions: Interactions debugger is now default](#addon-interactions-interactions-debugger-is-now-default)
   - [7.0 Vite changes](#70-vite-changes)
     - [Vite builder uses Vite config automatically](#vite-builder-uses-vite-config-automatically)
-    - [Vite cache moved to node_modules/.cache/.vite-storybook](#vite-cache-moved-to-node_modulescachevite-storybook)
+    - [Vite cache moved to node\_modules/.cache/.vite-storybook](#vite-cache-moved-to-node_modulescachevite-storybook)
   - [7.0 Webpack changes](#70-webpack-changes)
     - [Webpack4 support discontinued](#webpack4-support-discontinued)
     - [Babel mode v7 exclusively](#babel-mode-v7-exclusively)
@@ -167,7 +170,7 @@
     - [Angular: Drop support for calling Storybook directly](#angular-drop-support-for-calling-storybook-directly)
     - [Angular: Application providers and ModuleWithProviders](#angular-application-providers-and-modulewithproviders)
     - [Angular: Removed legacy renderer](#angular-removed-legacy-renderer)
-    - [Angular: initializer functions](#angular-initializer-functions)
+    - [Angular: Initializer functions](#angular-initializer-functions)
     - [Next.js: use the `@storybook/nextjs` framework](#nextjs-use-the-storybooknextjs-framework)
     - [SvelteKit: needs the `@storybook/sveltekit` framework](#sveltekit-needs-the-storybooksveltekit-framework)
     - [Vue3: replaced app export with setup](#vue3-replaced-app-export-with-setup)
@@ -206,7 +209,7 @@
     - [Dropped addon-docs manual babel configuration](#dropped-addon-docs-manual-babel-configuration)
     - [Dropped addon-docs manual configuration](#dropped-addon-docs-manual-configuration)
     - [Autoplay in docs](#autoplay-in-docs)
-    - [Removed STORYBOOK_REACT_CLASSES global](#removed-storybook_react_classes-global)
+    - [Removed STORYBOOK\_REACT\_CLASSES global](#removed-storybook_react_classes-global)
   - [7.0 Deprecations and default changes](#70-deprecations-and-default-changes)
     - [storyStoreV7 enabled by default](#storystorev7-enabled-by-default)
     - [`Story` type deprecated](#story-type-deprecated)
@@ -419,6 +422,32 @@
   - [Packages renaming](#packages-renaming)
   - [Deprecated embedded addons](#deprecated-embedded-addons)
 
+## From version 8.4.x to 8.5.x
+
+### Addon-a11y: Integration into `component tests`
+
+In Storybook 8.4, we have introduced a new addon called [addon test](https://storybook.js.org/docs/writing-tests/test-addon). This addon allows you to watch and run tests for all of your components in Storybook, featured by Vitest under the hood!
+
+In 8.5, we have revamped the `addon-a11y` to integrate with the component tests feature. Now, you can run a11y tests for your components in Storybook using the `addon-a11y` in component tests. If you upgrade to 8.5 via `npx storybook@latest upgrade`, the `addon-a11y` will be automatically integrated into your component tests. If you update to 8.5 manually and you already have `@storybook/experimental-addon-test` installed, please make the following changes to your `.storybook/vitest.setup.ts` file:
+
+```diff
+// .storybook/vitest.config.ts
+...
++import * as a11yAddonAnnotations from '@storybook/addon-a11y/preview';
+
+const annotations = setProjectAnnotations([
+  previewAnnotations,
++ a11yAddonAnnotations,
+]);
+
+// Run Storybook's beforeAll hook
+beforeAll(annotations.beforeAll);
+```
+
+### Addon-a11y: Deprecated `parameters.a11y.manual`
+
+We have deprecated `parameters.a11y.manual` in 8.5. Please use `globals.a11y.manual` instead.
+
 ## From version 8.2.x to 8.3.x
 
 ### Removed `experimental_SIDEBAR_BOTTOM` and deprecated `experimental_SIDEBAR_TOP` addon types

code/.storybook/preview.tsx

@@ -357,6 +357,9 @@ export const parameters = {
       opacity: 0.4,
     },
   },
+  a11y: {
+    warnings: ['minor', 'moderate', 'serious', 'critical'],
+  },
 };
 
 export const tags = ['test', 'vitest'];

code/.storybook/storybook.setup.ts

@@ -6,6 +6,8 @@ import { userEvent as storybookEvent, expect as storybookExpect } from '@storybo
 // eslint-disable-next-line import/namespace
 import * as testAnnotations from '@storybook/experimental-addon-test/preview';
 
+import * as a11yAddonAnnotations from '@storybook/addon-a11y/preview';
+
 import * as coreAnnotations from '../addons/toolbars/template/stories/preview';
 import * as componentAnnotations from '../core/template/stories/preview';
 // register global components used in many stories
@@ -15,9 +17,8 @@ import * as projectAnnotations from './preview';
 vi.spyOn(console, 'warn').mockImplementation((...args) => console.log(...args));
 
 const annotations = setProjectAnnotations([
-  // @ts-expect-error check type errors later
+  a11yAddonAnnotations,
   projectAnnotations,
-  // @ts-expect-error check type errors later
   componentAnnotations,
   coreAnnotations,
   testAnnotations,

code/addons/a11y/README.md

@@ -148,6 +148,30 @@ export const parameters = {
 };
 ```
 
+#### Overriding defaults
+
+Per default, Storybook has disabled the axe rule 'region' as it overreports in component testing.
+Landmarks are a good practice, but are usually applied during component composition, not on a component level. For example, a button itself doesn't define a landmark, whereas a whole page or a section of a page does.
+
+If you still want to enable the rule, you can do so by setting the `enabled` property to `true`:
+
+```ts
+// .storybook/preview.ts
+
+export const parameters = {
+  a11y: {
+    config: {
+      rules: [
+        {
+          id: 'region',
+          enabled: true,
+        },
+      ],
+    },
+  },
+};
+```
+
 ### Disabling checks
 
 If you wish to entirely disable `a11y` checks for a subset of stories, you can control this with story parameters:
@@ -161,9 +185,9 @@ MyNonCheckedStory.parameters = {
 };
 ```
 
-## Parameters
+## Parameters and globals
 
-For more customizability use parameters to configure [aXe options](https://github.com/dequelabs/axe-core/blob/develop/doc/API.md#api-name-axeconfigure).
+For more customizability use parameters and globals to configure [aXe options](https://github.com/dequelabs/axe-core/blob/develop/doc/API.md#api-name-axeconfigure).
 You can override these options [at story level too](https://storybook.js.org/docs/react/configure/features-and-behavior#per-story-options).
 
 ```js
@@ -180,10 +204,16 @@ export default {
       config: {},
       // axe-core optionsParameter (https://github.com/dequelabs/axe-core/blob/develop/doc/API.md#options-parameter)
       options: {},
-      // optional flag to prevent the automatic check
-      manual: true,
+      // Defines which impact levels will be considered as warnings instead of errors if executed via Storybook's component testing
+      warnings: <'minor' | 'moderate' | 'serious' | 'critical'>[]
     },
   },
+  globals: {
+    a11y: {
+      // optional flag to prevent the automatic check
+      manual: true,
+    }
+  }
 };
 
 export const accessible = () => <button>Accessible button</button>;
@@ -193,6 +223,82 @@ export const inaccessible = () => (
 );
 ```
 
+## Integration with Addon Test
+
+The a11y addon is compatible with Storybook's newest addon, the [addon test](https://storybook.js.org/docs/writing-tests/test-addon). When you run component tests, the a11y addon can automatically check for accessibility issues for all of your stories in the background. If there are any violations, the test will fail, and you will see the results in the sidebar.
+
+### Automatic setup
+
+When you add the a11y addon via `npx storybook add` command, it will automatically be integrated with the component testing feature. You don't need to do anything else to enable accessibility checks in your component tests.
+
+### Manual setup
+
+If you have already a11y addon installed and you have upgraded manually to Storybook 8.5 or later, you can enable the integration by adding the following configuration to your `.storybook/vitest.config.ts` file:
+
+```diff
+// .storybook/vitest.config.ts
+...
++import * as a11yAddonAnnotations from '@storybook/addon-a11y/preview';
+
+const annotations = setProjectAnnotations([
+  previewAnnotations,
++ a11yAddonAnnotations,
+]);
+
+// Run Storybook's beforeAll hook
+beforeAll(annotations.beforeAll);
+```
+
+### How to not get overwhelmed by a11y violations in addon-test
+
+Accessibility testing can be overwhelming, especially when you have a large number of stories and you have just started to use the a11y addon in combination with the test addon.
+
+You can disable accessibility tests which are running via the test addon for all of your stories at once. Just set the `!a11ytest` tag in your preview file:
+
+```js
+// .storybook/preview.js
+export const tags = ['!a11ytest'];
+```
+
+Now you can step-by-step enable accessibility tests for your stories by adding the `a11ytest` tag to the stories' meta or by adding the `a11ytest` tag to the story's tags directly:
+
+```js
+// Button.stories.js
+export default {
+  title: 'Button',
+  component: Button,
+  // add the tag to the meta to enable accessibility checks for all of your stories
+  tags: ['a11ytest'],
+};
+
+export const Primary = {
+  // add the tag to story itself to just enable the accessibility check for this story
+  tags: ['a11ytest'],
+  ...
+}
+```
+
+### Violation impact levels
+
+By default, the addon will consider all violations as errors. However, you can configure the addon to consider some violations as warnings instead of errors. This can be useful when `@storybook/addon-a11y` is used in combination with `@storybook/experimental-addon-test`. To do this, set the `warnings` parameter in the `a11y` object to an array of impact levels that should be considered as warnings.
+
+```js
+export default {
+  title: 'button',
+  parameters: {
+    a11y: {
+      /**
+       * @default [ ]
+       * @type: Array<'minor' | 'moderate' | 'serious' | 'critical'>
+       */
+      warnings: ['minor', 'moderate'],
+    },
+  },
+};
+```
+
+In this example, all violations with an impact level of `minor` or `moderate` will be considered as warnings. All other violations will be considered as errors. When running automated UI tests featured by Vitest, all violations with an impact level of `serious` or `critical` will now fail the test. This failure is reflected as an error in the sidebar or when running Vitest separately. `minor` and `moderate` violations will be reported as warnings but will not fail the test.
+
 ## Automate accessibility tests with test runner
 
 The test runner does not apply any rules that you have set on your stories by default. You can configure the runner to correctly apply the rules by [following the guide on the Storybook docs](https://storybook.js.org/docs/writing-tests/accessibility-testing#automate-accessibility-tests-with-test-runner).

code/addons/a11y/package.json

@@ -56,12 +56,16 @@
   },
   "dependencies": {
     "@storybook/addon-highlight": "workspace:*",
-    "axe-core": "^4.2.0"
+    "@storybook/test": "workspace:*",
+    "axe-core": "^4.2.0",
+    "vitest-axe": "^0.1.0"
   },
   "devDependencies": {
     "@storybook/global": "^5.0.0",
     "@storybook/icons": "^1.2.12",
     "@testing-library/react": "^14.0.0",
+    "picocolors": "^1.1.0",
+    "pretty-format": "29.7.0",
     "react": "^18.2.0",
     "react-dom": "^18.2.0",
     "react-resize-detector": "^7.1.2",

code/addons/a11y/src/a11yRunner.test.ts

@@ -22,7 +22,6 @@ describe('a11yRunner', () => {
     await import('./a11yRunner');
 
     expect(mockedAddons.getChannel).toHaveBeenCalled();
-    expect(mockChannel.on).toHaveBeenCalledWith(EVENTS.REQUEST, expect.any(Function));
     expect(mockChannel.on).toHaveBeenCalledWith(EVENTS.MANUAL, expect.any(Function));
   });
 });