sphinx-build exit code is 0 with ERRORS

[g-chauvel]

Describe the bug

malformed table errors are reported as ERROR when running sphinx-build

source\a\a.rst:14: ERROR: Error parsing content block for the "list-table" directive: uniform two-level bullet list expected, but row 3 does not contain the same number of items as row 1 (4 vs 5).
source\a\a.rst:35: ERROR: "list-table" widths do not match the number of columns in table (5).
source\a\a.rst:51: ERROR: "list-table" widths do not match the number of columns in table (5).

but the exit code is 0

How to Reproduce

$ git clone https://github.com/g-chauvel/sphinx-doc
$ cd sphinx-doc
$ sphinx-build source build

Expected behavior

exit code is different from 0

Your project

https://github.com/g-chauvel/sphinx-doc

Screenshots

No response

OS

Windows 10

Python version

3.9.12

Sphinx version

4.5.0

Sphinx extensions

No response

Extra tools

No response

Additional context

No response

[tk0miya]

Could you try -W option, please?

[g-chauvel]

I had.
Warning, treated as error: , it stops on first one

[tk0miya]

If you'd like to see the all of warnings (like CI), please use -W --keep-going. Then you'll see all warnings and errors, and will get the non-zero return code.

[revati-naik]

I had a question on this one, what if there are a few WARNINGS which should not be treated as errors, in that case, how do we differentiate?

[aeisenbarth]

I had a question on this one, what if there are a few WARNINGS which should not be treated as errors, in that case, how do we differentiate?

This is a very good question! We also have too many warnings that are irrelevant, that we cannot control (#11325) or that are false positives.
I was assuming I could skip reporting non-error-level warnings with -Q (--silent) and raise those of level ERROR with -W (--fail-on-warning). Apparently, this does not work (and it would not be good design). It would be good to have an option like --fail-on-warning-level=ERROR for which severity level should cause a failure, or an intuitive regular expression filter for which exact warnings should fail.

In the end, I found for myst_nb implements an extra nb_execution_raise_on_error. They must have had a reason (executablebooks/MyST-NB#248) to build this despite Sphinx's mechanism.