Record exception backtraces during test suite runs #317

craigfe · 2021-08-16T13:17:44Z

craigfe · 2021-08-19T20:03:59Z

The 4.12+domains CI is consistently failing to run the Lwt examples binary on this PR, failing with the following segfault:

Starting program: /home/craigfe/t/alcotest/_build/default/examples/lwt/test.exe
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Testing `LwtUtils'.
This run has ID `WQI2PGWL'.

  [OK]          basic               0   Plain.
  [OK]          basic               1   Lwt.
  [OK]          exceptions          0   Plain.
  [OK]          exceptions          1   Lwt toplevel.
  [OK]          exceptions          2   Lwt internal.
  [OK]          switches            0   Allocate resource.
  [OK]          switches            1   Check resource deallocated.

Full test results in `~/t/alcotest/_build/default/examples/lwt/_build/_tests/LwtUtils'.
Test Successful in 0.002s. 7 tests run.

Program received signal SIGSEGV, Segmentation fault.
caml_darken (v=0, ignored=<optimized out>, state=<optimized out>) at major_gc.c:760
760	in major_gc.c
(gdb) bt
#0  caml_darken (v=0, ignored=<optimized out>, state=<optimized out>) at major_gc.c:760
#1  0x0000555555749fd6 in caml_iterate_global_roots (rootlist=0x55555585ae60 <caml_global_roots_old>, rootlist=0x55555585ae60 <caml_global_roots_old>, fdata=0x0, f=0x5555557307a0 <caml_darken>) at globroots.c:222
#2  caml_scan_global_roots (f=f@entry=0x5555557307a0 <caml_darken>, fdata=fdata@entry=0x0) at globroots.c:233
#3  0x0000555555731257 in cycle_all_domains_callback (domain=domain@entry=0x7ffff7b80000, unused=unused@entry=0x0, participating_count=<optimized out>, participating=participating@entry=0x555555864c40 <stw_request+64>) at major_gc.c:1092
#4  0x0000555555752540 in caml_try_run_on_all_domains_with_spin_work (handler=handler@entry=0x555555731180 <cycle_all_domains_callback>, data=data@entry=0x0, leader_setup=leader_setup@entry=0x0,
    enter_spin_callback=enter_spin_callback@entry=0x0, enter_spin_data=enter_spin_data@entry=0x0) at domain.c:977
#5  0x000055555575261a in caml_try_run_on_all_domains (handler=handler@entry=0x555555731180 <cycle_all_domains_callback>, data=data@entry=0x0, leader_setup=leader_setup@entry=0x0) at domain.c:991
#6  0x0000555555731fae in major_collection_slice (howmuch=<optimized out>, participant_count=participant_count@entry=0, barrier_participants=barrier_participants@entry=0x0, mode=mode@entry=Slice_interruptible) at major_gc.c:1376
#7  0x0000555555732138 in caml_major_collection_slice (howmuch=howmuch@entry=-1) at major_gc.c:1395
#8  0x0000555555750db5 in caml_poll_gc_work () at domain.c:1041
#9  0x0000555555734844 in caml_alloc (tag=<optimized out>, wosize=<optimized out>) at alloc.c:43
#10 caml_alloc (wosize=<optimized out>, tag=<optimized out>) at alloc.c:33
#11 0x000055555573c9e9 in caml_ml_out_channels_list (unit=<optimized out>) at io.c:568
#12 <signal handler called>
#13 camlStdlib__flush_all_232 () at stdlib.ml:350
#14 0x00005555556cd966 in camlStdlib__exit_475 () at stdlib.ml:561
#15 0x000055555566aa66 in camlAlcotest_engine__Monad__fun_400 () at src/alcotest-engine/monad.ml:32
#16 0x00005555556b3ee8 in camlCmdliner_term__fun_186 () at cmdliner_term.ml:25
#17 0x00005555556b7b59 in camlCmdliner__run_418 () at cmdliner.ml:117
#18 0x00005555556b7e98 in camlCmdliner__term_eval_444 () at cmdliner.ml:147
#19 0x00005555556b8c22 in camlCmdliner__eval_choice_inner_1653 () at cmdliner.ml:265
#20 0x000055555567807a in camlAlcotest_engine__Cli__run_with_args$27_1097 () at src/alcotest-engine/cli.ml:108
#21 0x0000555555645765 in camlDune__exe__Test__entry () at examples/lwt/test.ml:82
#22 0x0000555555640756 in caml_program ()
#23 <signal handler called>
#24 0x000055555572e947 in caml_startup_common (argv=0x7fffffffd6c8, pooling=<optimized out>, pooling@entry=0) at startup_nat.c:133
#25 0x000055555572e978 in caml_startup_exn (argv=<optimized out>) at startup_nat.c:137
#26 caml_main (argv=<optimized out>) at startup_nat.c:142
#27 0x000055555563ff4c in main (argc=<optimized out>, argv=<optimized out>) at main.c:41

The segfault happens at the very end of running the binary, during flushing on the call to exit 0. Not sure what to make of this, but it seems unrelated to the PR. @ctk21: any thoughts?

Fix mirage#315.

The number of lines in the stack trace of a failure is compiler-dependent, so it should not be captured in the test cases.

craigfe · 2021-10-08T16:56:40Z

Rebased and fixed the expect tests to account for changes made in #326.

At time of writing, this varies according to the version of `lwt` that is installed, so we can't snapshot it in the expect tests.

craigfe · 2021-10-08T19:25:05Z

FTR, I tried to avoid duplicating snapshot outputs between the native and JS expect tests (here), but couldn't find a diff that behaves consistently on all three platforms. (Unfortunately, it seems there's no patdiff available for a modern version of OCaml on our Windows CI runner.) Just doing with the duplicated snapshots for now.

Merging now as the CI is happy.

@hhugo

…and alcotest-lwt (1.5.0) CHANGES: - Make Alcotest compatible with `js_of_ocaml.3.11.0`. Users can depend on the new virtual `alcotest-js` Opam library to pick up the right `js_of_ocaml` version automatically. (mirage/alcotest#326 mirage/alcotest#328, @hhugo @smorimoto) - Record exception backtraces during test suite runs by default. This behaviour can be disabled by passing `~record_backtrace:false` to `Alcotest.run`. (mirage/alcotest#317, @craigfe) - Generate shorter unique identifiers for test runs (8-character alphanumeric, rather than a full 128-bit UUID). (mirage/alcotest#304, @craigfe) - Change `Alcotest.{char,string}` pretty-printers to use OCaml syntax on assertion failures (i.e. wrap with quotes and escape control characters). (mirage/alcotest#318, @craigfe) - Fix process for getting the width of attached terminals on MacOS. Previously, a terminal width of 80 columns was assumed. (mirage/alcotest#325, @craigfe) - Fix parsing of test filter ranges to allow '-' separators (e.g. `test alpha 1-4`), as advertised in the manpage. The previously-used '..' separator is also supported. (mirage/alcotest#312, @craigfe) - Introduce an `Alcotest.V1` module that aliases the existing `Alcotest` API and provides a stability guarantee over major version changes. Similar versioned aliases also exist for the backends: `Alcotest_{async,lwt}.V1`. (mirage/alcotest#306, @craigfe) - Change the `~filter` argument to `Alcotest.run` to be a predicate over tests. (mirage/alcotest#305, @craigfe) - Renamed / removed some less frequently used modules used by the test backends: - `Alcotest.Unix` -> `Alcotest.Unix_platform` - `Alcotest_engine.{Cli,Core,Test}` -> `Alcotest_engine.V1.{Cli,Core,Test}` - `Alcotest.{Cli,Core}` are now gone. Use `Alcotest_engine.V1.{Cli,Core}.Make (Alcotest.Unix_platform)` instead. (mirage/alcotest#306 mirage/alcotest#309, @craigfe) - Avoid exporting `list_tests` in the main test APIs (`Alcotest{,_lwt,_async}`). Use `Alcotest_engine` directly if you want this function. (mirage/alcotest#310, @craigfe)

@hhugo

…and alcotest-lwt (1.5.0) CHANGES: - Make Alcotest compatible with `js_of_ocaml.3.11.0`. Users can depend on the new virtual `alcotest-js` Opam library to pick up the right `js_of_ocaml` version automatically. (mirage/alcotest#326 mirage/alcotest#328, @hhugo @smorimoto) - Record exception backtraces during test suite runs by default. This behaviour can be disabled by passing `~record_backtrace:false` to `Alcotest.run`. (mirage/alcotest#317, @craigfe) - Generate shorter unique identifiers for test runs (8-character alphanumeric, rather than a full 128-bit UUID). (mirage/alcotest#304, @craigfe) - Change `Alcotest.{char,string}` pretty-printers to use OCaml syntax on assertion failures (i.e. wrap with quotes and escape control characters). (mirage/alcotest#318, @craigfe) - Fix process for getting the width of attached terminals on MacOS. Previously, a terminal width of 80 columns was assumed. (mirage/alcotest#325, @craigfe) - Fix parsing of test filter ranges to allow '-' separators (e.g. `test alpha 1-4`), as advertised in the manpage. The previously-used '..' separator is also supported. (mirage/alcotest#312, @craigfe) - Introduce an `Alcotest.V1` module that aliases the existing `Alcotest` API and provides a stability guarantee over major version changes. Similar versioned aliases also exist for the backends: `Alcotest_{async,lwt}.V1`. (mirage/alcotest#306, @craigfe) - Change the `~filter` argument to `Alcotest.run` to be a predicate over tests. (mirage/alcotest#305, @craigfe) - Renamed / removed some less frequently used modules used by the test backends: - `Alcotest.Unix` -> `Alcotest.Unix_platform` - `Alcotest_engine.{Cli,Core,Test}` -> `Alcotest_engine.V1.{Cli,Core,Test}` - `Alcotest.{Cli,Core}` are now gone. Use `Alcotest_engine.V1.{Cli,Core}.Make (Alcotest.Unix_platform)` instead. (mirage/alcotest#306 mirage/alcotest#309, @craigfe) - Avoid exporting `list_tests` in the main test APIs (`Alcotest{,_lwt,_async}`). Use `Alcotest_engine` directly if you want this function. (mirage/alcotest#310, @craigfe)

hhugo · 2021-10-12T09:10:43Z

FTR, I tried to avoid duplicating snapshot outputs between the native and JS expect tests (here), but couldn't find a diff that behaves consistently on all three platforms. (Unfortunately, it seems there's no patdiff available for a modern version of OCaml on our Windows CI runner.) Just doing with the duplicated snapshots for now.

Merging now as the CI is happy.

The major downside is that it's now easier to introduce a change behaving differently in native and js

craigfe · 2021-10-12T09:13:12Z

Agreed. If I'd found a way to do reproducible diffing, I'd have gone with that instead. For now, it'll be necessary to run some diff tool manually when tests are added.

craigfe force-pushed the raise-backtrace branch 4 times, most recently from f08632a to 56a7c22 Compare August 19, 2021 18:28

craigfe assigned Ngoguey42 Aug 23, 2021

Ngoguey42 approved these changes Aug 25, 2021

View reviewed changes

craigfe unassigned Ngoguey42 Aug 25, 2021

craigfe force-pushed the raise-backtrace branch from 56a7c22 to 92ef81e Compare October 8, 2021 13:01

craigfe added 2 commits October 8, 2021 14:09

Record exception backtraces during test suite runs

2ec6c7b

Fix mirage#315.

Add support for record_backtrace parameter to Alcotest.run

52ca007

craigfe force-pushed the raise-backtrace branch 5 times, most recently from 5583769 to b896883 Compare October 8, 2021 16:45

craigfe added 2 commits October 8, 2021 17:55

test: avoid printing number of stacktrace lines in expect tests

15f6b61

The number of lines in the stack trace of a failure is compiler-dependent, so it should not be captured in the test cases.

test: allow JavaScript expect tests to not emit a stacktrace

c9eb92d

craigfe force-pushed the raise-backtrace branch from b896883 to 14ac451 Compare October 8, 2021 16:55

craigfe force-pushed the raise-backtrace branch from 49e6a18 to 23b91f4 Compare October 8, 2021 18:39

test: don't depend on Lwt stacktrace printing

aef3ebe

At time of writing, this varies according to the version of `lwt` that is installed, so we can't snapshot it in the expect tests.

craigfe force-pushed the raise-backtrace branch from 23b91f4 to aef3ebe Compare October 8, 2021 19:02

craigfe merged commit a37d41e into mirage:master Oct 8, 2021

craigfe mentioned this pull request Oct 9, 2021

[new release] alcotest, alcotest-async, alcotest-js, alcotest-mirage and alcotest-lwt (1.5.0) ocaml/opam-repository#19735

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Record exception backtraces during test suite runs #317

Record exception backtraces during test suite runs #317

craigfe commented Aug 16, 2021

craigfe commented Aug 19, 2021

craigfe commented Oct 8, 2021

craigfe commented Oct 8, 2021

hhugo commented Oct 12, 2021

craigfe commented Oct 12, 2021

Record exception backtraces during test suite runs #317

Record exception backtraces during test suite runs #317

Conversation

craigfe commented Aug 16, 2021

craigfe commented Aug 19, 2021

craigfe commented Oct 8, 2021

craigfe commented Oct 8, 2021

hhugo commented Oct 12, 2021

craigfe commented Oct 12, 2021