Skip to content

Commit

Permalink
axe api changes with more fine grained exceptions (#26)
Browse files Browse the repository at this point in the history
* axe api changes with more fine grained exceptions

* raises correct exception

* raises type or tuple of types

* update settings test

* move exceptions to their own file

* add hypothesis to gitignore

* update git for pixi info

* update locks

* add pixi

* add a workflow that uses pixi

* control ff install location

* updates tests and dependencies

* pad cell numbers

* exclude widegets from settings test

* fix settings error

* include window sin the tests

* rm raise expr

* update pixi lock

* skip ubuntu windows

* update pixi

* add docs to pixi

* fix bad raise in test

* update the artifact downloader

* add scripts for extracting axe rules

* mv axe and testing utils

* update tests

* stricter test conditions

* change w3c fixtures

* bug in test

* update tests

* fix test and deprecation warning

* remove passing test

* updating tests to satisfy ci

* drop old non pixi workflow

* rnm pixi test workflow

* squash old code

* add generate function for testing axe

* add a test for the axe test tools
  • Loading branch information
tonyfast authored Jul 10, 2024
1 parent bcabb11 commit 88aef22
Show file tree
Hide file tree
Showing 35 changed files with 11,977 additions and 558 deletions.
3 changes: 3 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# GitHub syntax highlighting
pixi.lock linguist-language=YAML

109 changes: 109 additions & 0 deletions .github/workflows/pixi.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
name: test with pixi

on:
push:
branches: [main]
pull_request:
branches: ["*"]
workflow_dispatch:

concurrency:
group: ${{ github.ref }}
cancel-in-progress: true

env:
PYTHONUNBUFFERED: "1"
PIP_DISABLE_PIP_VERSION_CHECK: "1"
USE_PIXI_VERSION: 0.22.0

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: prefix-dev/setup-pixi@v0.6.0
with:
pixi-version: v${{ env.USE_PIXI_VERSION }}
cache: true
environments: build
- run: pixi run -ve build build
- uses: actions/upload-artifact@v4
with:
name: nbconvert-a11y-${{ github.run_number }}-dist
path: ./dist

test:
name: (${{ matrix.os }}) test
needs: [build]
runs-on: ${{ matrix.os }}-latest
strategy:
fail-fast: false
matrix:
os: ["ubuntu"]
# add matrix for browsers later on
steps:
- run: git config --global core.autocrlf false
- uses: actions/checkout@v4
- uses: prefix-dev/setup-pixi@v0.6.0
with:
pixi-version: v${{ env.USE_PIXI_VERSION }}
cache: true
environments: test
- uses: actions/download-artifact@v4
with:
name: nbconvert-a11y-${{ github.run_number }}-dist
path: ./dist
# need to make this install from the wheel
- run: pixi run -ve test test
- if: always()
uses: actions/upload-artifact@v4
with:
name: |-
nbconvert-a11y-${{ github.run_number }}-${{ matrix.os }}-test
path: |
./.pixi/task-cache-v0
./build/reports
docs:
name: docs
needs: [test]
runs-on: ubuntu-latest
steps:
- run: git config --global core.autocrlf false
- uses: actions/checkout@v4
- uses: prefix-dev/setup-pixi@v0.6.0
with:
pixi-version: v${{ env.USE_PIXI_VERSION }}
cache: true
environments: docs
- run: pixi run -ve docs build
- if: always()
uses: actions/upload-artifact@v4
with:
name: nbconvert-a11y-${{ github.run_number }}-site
path: site

publish:
name: publish the mkdocs build to github pages
needs: [docs]
runs-on: ubuntu-latest
steps:
- name: checkout repo
uses: actions/checkout@v4
- uses: actions/download-artifact@v4
with:
name: nbconvert-a11y-${{ github.run_number }}-site
path: site
- name: Deploy main 🚀
uses: JamesIves/github-pages-deploy-action@v4
if: ${{ github.ref_name == 'main' }}
with:
folder: site # The folder the action should deploy.
single-commit: true
- name: Deploy non-main 🚀
uses: JamesIves/github-pages-deploy-action@v4
if: ${{ github.ref_name != 'main' }}
with:
folder: site # The folder the action should deploy.
single-commit: true
target-folder: branch/${{ github.ref_name }}
188 changes: 64 additions & 124 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -1,116 +1,98 @@
name: pytest nbconvert-a11y, axe test exports, build docs.
name: test with pixi

on:
push:
branches: [main]
pull_request:
branches: ['*']
branches: ["*"]
workflow_dispatch:

concurrency:
group: ${{ github.ref }}
cancel-in-progress: true

env:
# Increase this value to reset cache if environments have not changed
CACHE_NUMBER: 2
# squash some known warnings
JUPYTER_PLATFORM_DIRS: 1
PYTHONUNBUFFERED: "1"
PIP_DISABLE_PIP_VERSION_CHECK: "1"
USE_PIXI_VERSION: 0.22.0

jobs:
format:
name: format
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: prefix-dev/setup-pixi@v0.6.0
with:
fetch-depth: 0
- uses: actions/setup-python@v4
pixi-version: v${{ env.USE_PIXI_VERSION }}
cache: true
environments: build
- run: pixi run -ve build build
- uses: actions/upload-artifact@v4
with:
python-version: "3.12"
cache: pip
cache-dependency-path: pyproject.toml
- name: install dev dependencies
run: python -m pip install --upgrade pip hatch
- name: run formatters
run: |
echo "~~~bash" > "${GITHUB_STEP_SUMMARY}"
hatch run format:code 2>&1 | tee --append "${GITHUB_STEP_SUMMARY}"
echo "~~~" >> "${GITHUB_STEP_SUMMARY}"
- name: print diff
run: |
echo "~~~diff" >> "${GITHUB_STEP_SUMMARY}"
git diff | tee --append "${GITHUB_STEP_SUMMARY}"
echo "~~~" >> "${GITHUB_STEP_SUMMARY}"
name: nbconvert-a11y-${{ github.run_number }}-dist
path: ./dist

test:
name: test package and accessibility
defaults:
run:
shell: bash -el {0}
name: (${{ matrix.os }}) test
needs: [build]
runs-on: ${{ matrix.os }}-latest
strategy:
fail-fast: false
matrix:
python-version:
- "3.10"
runs-on: ubuntu-latest
os: ["ubuntu"]
# add matrix for browsers later on
steps:
- name: fetch all history and tags
uses: actions/checkout@v3
- run: git config --global core.autocrlf false
- uses: actions/checkout@v4
- uses: prefix-dev/setup-pixi@v0.6.0
with:
fetch-depth: 0
- uses: mamba-org/setup-micromamba@v1
pixi-version: v${{ env.USE_PIXI_VERSION }}
cache: true
environments: test
- uses: actions/download-artifact@v4
with:
environment-file: .github/test-environment.yml
- name: init plawright
run: |
playwright install --with-deps chromium
- name: init node & files
run: |
yarn
doit copy
- name: init dev module
run: |
python3 -m pip install -e . --no-deps --no-build-isolation --ignore-installed
- name: check pip env
run: |
python3 -m pip check
- name: smoke test
run: |
# the smoke generate html assets that are used in the accessibility testing.
# we run this script to generate assets and test the nbconvert-a11y module.
# failures here will stop any docs builds
pytest --color=yes tests/test_smoke.py
- name: build wheel and sdist
run: |
pyproject-build
- uses: actions/upload-artifact@v3
name: nbconvert-a11y-${{ github.run_number }}-dist
path: ./dist
# need to make this install from the wheel
- run: pixi run -ve test test
- if: always()
uses: actions/upload-artifact@v4
with:
name: dist
path: dist
- name: mkdocs
run: |
mkdocs build -v
- uses: actions/upload-artifact@v3
name: |-
nbconvert-a11y-${{ github.run_number }}-${{ matrix.os }}-test
path: |
./.pixi/task-cache-v0
./build/reports
docs:
name: docs
needs: [test]
runs-on: ubuntu-latest
steps:
- run: git config --global core.autocrlf false
- uses: actions/checkout@v4
- uses: prefix-dev/setup-pixi@v0.6.0
with:
pixi-version: v${{ env.USE_PIXI_VERSION }}
cache: true
environments: docs
- run: pixi run -ve docs build
- if: always()
uses: actions/upload-artifact@v4
with:
name: site
name: nbconvert-a11y-${{ github.run_number }}-site
path: site
- name: a11y tests
# always build the docs to see what the new versions look like.
# continue-on-error: true
run: |
pytest \
--color=yes \
-n auto \
--deselect tests/test_smoke.py \
--self-contained-html \
--html=tests/exports/pytest/report.html

publish:
name: publish the mkdocs build to github pages
needs: [test]
needs: [docs]
runs-on: ubuntu-latest
steps:
- name: checkout repo
uses: actions/checkout@v3
- uses: actions/download-artifact@v3
uses: actions/checkout@v4
- uses: actions/download-artifact@v4
with:
name: site
name: nbconvert-a11y-${{ github.run_number }}-site
path: site
- name: Deploy main 🚀
uses: JamesIves/github-pages-deploy-action@v4
Expand All @@ -125,45 +107,3 @@ jobs:
folder: site # The folder the action should deploy.
single-commit: true
target-folder: branch/${{ github.ref_name }}

release:
name: draft release when tagged
if: startsWith(github.ref, 'refs/tags/')
needs: [test]
runs-on: ubuntu-latest
permissions:
id-token: write
contents: write
steps:
- name: fetch contents
uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with:
name: dist
path: dist
- uses: actions/setup-python@v4
with:
python-version: "3.11"
cache: pip
cache-dependency-path: pyproject.toml
- name: install twine and pytest
run: |
pip install twine pytest
- name: Publish package distributions to TestPyPI
run: |
twine upload --repository testpypi \
--user __token__ --password ${{secrets.HATCH_TEST_INDEX_AUTH}} \
dist/*
- name: install nbconvert-a11y dependencies from test pip
run: |
pip install \
--index-url 'https://test.pypi.org/simple/' \
--extra-index-url 'https://pypi.org/simple/' \
nbconvert-a11y
- name: test test release
run: |
pytest tests/test_smoke.py
- uses: ncipollo/release-action@v1
with:
artifacts: "dist/.*"
draft: true # does not trigger a created event
8 changes: 7 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
.hypothesis
.pixi
.local-browsers


dist/
__pycache__
.#*
.doit.*
Expand All @@ -20,4 +26,4 @@ settings.json
site
tests/exports
tests/out.html
tests/outputs/*.html
tests/outputs/*.html
27 changes: 27 additions & 0 deletions nbconvert_a11y/axe/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# testing axe in python

the `test_axe` module provides utilities for running `axe-core` on html document in python. it is designed for asynchronous interactive use and synchronous testing with the `pytest_axe` extension.

`axe-core` is a javascript module that must be run inside of a live browser. currently, `playwright` is the only browser automation supporter, but `selenium` could be added.

<!-- can we add it to robots from jupyter after that? -->

## axe exceptions in python

each release contains a version of axe and a set of exceptions generated for that specific version.
the types are useful in accounting for expected accessibility that can be fixed later.

## playwright testing

it turns out the best ergonomics with working with axe from playwright are to append methods to the `Page` class. it has the following methods when `test_axe` is active:

`Page.axe`
: run axe and return the test results as a python dictionary

`Page.test_axe`
: run axe and return the test results as a python exception group

`Page.aom`
: an alias to the soon to be deprecated `page.accessibility.snapshot()`

we have to main synchronous and asynchronous methods for playwright
4 changes: 4 additions & 0 deletions nbconvert_a11y/axe/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from .async_axe import *
from ..browsers import Browser
from .axe_exceptions import *
from .ipy import load_ipython_extension, unload_ipython_extension
Loading

0 comments on commit 88aef22

Please sign in to comment.