Assertion of category is not working

[mkathu]

Describe the bug
On triggering the scan assertion is not done at the category.

To Reproduce
Steps to reproduce the behavior:

1. Configure the page to be scanned
2. Specify the minimum performance score and other metrics for assertion
3. Execute the scan for the page
4. Open the report

Expected behavior
Once execution is done assertion should be done should pass/fail based on configured values

Observed:
The assertion is not done for category but metrics

Logs/Screenshots

Environment (please complete the following information):

• OS: Ubuntu latest
• Browser : chrome

[volodya315]

I get the same result with assert matrix.
For both urls assertions should fail, but everything passes

        assert: {
            assertMatrix: [
                {
                    matchingUrlPattern: "url1",
                    assertions: {
                        'categories:performance': ['error', { minScore: 0.99 }],
                    }
                },
                {
                    matchingUrlPattern: "url2",
                    assertions: {
                        'categories:performance': ['error', { minScore: 0.99 }],
                    }
                }
            ]
        },

It only works if I replace assertMatrix with one assert for every url, but I need to set different categories scores for different environments

      assert: {
            assertions: {
                'categories:performance': ['error', { minScore: 0.9 }],
            },

OS: MacOS 12.0.1
Browser: Chrome

[wac2007]

Some updates on this? I'm having the same problem :/

[svobom57]

I think I'm experiencing the same issue sometimes on my CI. Is it possible it has to do with how Lighthouse runs the assertions?

In the docs there is a section mentioning the aggregation of the results:
https://github.com/GoogleChrome/lighthouse-ci/blob/main/docs/configuration.md#aggregation-methods
and it seems like the default is optimistic whatever it means. Could this be the issue?