video-caption accessibility audit always shows as passing even in error scenarios

[thibaudcolas]

FAQ

• Yes, my issue is not about variability or throttling.
• [X] Yes, my issue is not about a specific accessibility audit (file with axe-core instead).

URL

https://accessibility-experiments-90d5b5.glitch.me/lighthouse-video-caption-audit/index.html

What happened?

Same issue as #14438 and #14354, this time for video-caption. This is the last rule I was suspecting to have this problem!

The video-caption audit based on Axe’s video-caption rule always shows as passing when it’s applicable, even when Axe itself reports an "incomplete" / "needs review" result.

Sample page: https://accessibility-experiments-90d5b5.glitch.me/lighthouse-video-caption-audit/index.html

Lighthouse results for video-caption on this page (JSON export)

"video-caption": {
      "id": "video-caption",
      "title": "`<video>` elements contain a `<track>` element with `[kind=\"captions\"]`",
      "description": "When a video provides a caption it is easier for deaf and hearing impaired users to access its information. [Learn more](https://web.dev/video-caption/).",
      "score": 1,
      "scoreDisplayMode": "binary",
      "details": {
        "type": "table",
        "headings": [],
        "items": []
      }
    },

Axe CLI results for the same page

Command:

axe -r video-caption https://accessibility-experiments-90d5b5.glitch.me/lighthouse-video-caption-audit/index.html --save video-caption.json && cat video-caption.json

Output:

[
  {
    "testEngine": {
      "name": "axe-core",
      "version": "4.4.3"
    },
    "testRunner": {
      "name": "axe"
    },
    "testEnvironment": {
      "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/105.0.5195.125 Safari/537.36",
      "windowWidth": 800,
      "windowHeight": 600,
      "orientationAngle": 0,
      "orientationType": "landscape-primary"
    },
    "timestamp": "2022-10-06T23:30:17.103Z",
    "url": "https://accessibility-experiments-90d5b5.glitch.me/lighthouse-video-caption-audit/index.html",
    "toolOptions": {
      "runOnly": {
        "type": "rule",
        "values": [
          "video-caption"
        ]
      },
      "reporter": "v1"
    },
    "inapplicable": [],
    "passes": [],
    "incomplete": [
      {
        "id": "video-caption",
        "impact": "critical",
        "tags": [
          "cat.text-alternatives",
          "wcag2a",
          "wcag122",
          "section508",
          "section508.22.a"
        ],
        "description": "Ensures <video> elements have captions",
        "help": "<video> elements must have captions",
        "helpUrl": "https://dequeuniversity.com/rules/axe/4.4/video-caption?application=webdriverjs",
        "nodes": [
          {
            "any": [],
            "all": [],
            "none": [
              {
                "id": "caption",
                "data": null,
                "relatedNodes": [],
                "impact": "critical",
                "message": "Check that captions is available for the element"
              }
            ],
            "impact": "critical",
            "html": "<video width=\"300\" height=\"200\">\n    <source src=\"myVideo.mp4\" type=\"video/mp4\">\n</video>",
            "target": [
              "video"
            ],
            "failureSummary": "Fix all of the following:\n  Check that captions is available for the element"
          }
        ]
      }
    ],
    "violations": []
  }
]

What did you expect?

Lighthouse’s accessibility audit results should be consistent with those of Axe. In this instance, Axe reports an "incomplete", so it shouldn’t be a pass with Lighthouse. I would have expected either a "Additional items to manually check", or an error.

What have you tried?

Aside from setting up the demo page to reproduce this definitively, I also identified the Axe change that likely caused this – switching the rule from reporting errors to always reporting incomplete: axe-core#285, dequelabs/axe-core@30be884.

How were you running Lighthouse?

Chrome DevTools

Lighthouse Version

9.6.6

Chrome Version

108

Node Version

No response

OS

Mac

Relevant log output

No response

[adamraine]

Merging into #14354