Are all tests expected to pass?

[norbertwenzel]

I've simply cloned the repo (master & v1.3.6) and used CMake to build and run the tests. But one test fails on consistently on all platforms (Linux, Mac, Windows). I could not find any open issues about a failing test therefore I thought I'd ask whether this is known and/or to be expected.

Steps to reproduce:

Clone repository and change to project's root folder. In this folder run:

cmake -H. -Bbuild
cmake --build build -- -j
cmake -E env CTEST_OUTPUT_ON_FAILURE=1 cmake --build build --target test

Result:

96% tests passed, 1 tests failed out of 24

Total Test time (real) =   0.54 sec

The following tests FAILED:
	 16 - tlmdifc (Failed)
Errors while running CTest
Makefile:108: recipe for target 'test' failed
make: *** [test] Error 8

The test output written to build/examples/tlmdifc.out is:

 final l2 norm of the residuals     0.09063596

 number of function evaluations        21

 exit parameter                         1

 final approximate solution

     0.08241058       1.133037       2.343695
 covariance

   0.0001531202    0.002869942   -0.002656662
    0.002869942     0.09480937    -0.09098997
   -0.002656662    -0.09098997     0.08778729

These results are from a run on Ubuntu Bionic amd64 but the very same test also failed for me on Mac and Windows.

[devernay]

Take a look at the difference with the reference file https://github.com/devernay/cminpack/blob/master/examples/ref/tlmdifc.ref .
In this case, the second and third column of the cov matrix are slightly different.
I've seen floating point computation, especially when optimized, behave slightly differently on different OSs/architectures. If the difference is non-significative, then you should consider that the tests pass.
I agree that the tests should be rewritten to handle that, but "diff" was all I had when the tests were written.

[StefanBruens]

For other architectures (aarch64, ppc64le, ...) every second test fails due to numeric differences.

[devernay]

Yes, this is a known issue, feel free to contribute a better est validation that using diff on the text output

[devernay]

I've been experimenting with numdiff, it's not ideal either. Let's say that the tests are supposed to pass on CI platforms, but on your own machine YMMV, especially if you use BLAS.

[devernay]

There is a better MINPACK test suite in https://github.com/cbouilla/minpack-1.1 (see cbouilla/minpack-1.1#1)