Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression Test #23

Closed
wants to merge 132 commits into from
Closed

Regression Test #23

wants to merge 132 commits into from

Conversation

ghost
Copy link

@ghost ghost commented May 4, 2017

This pull request adds a simple regression test borrowing heavily from @jrood-nrel's Nalu regression test and @gantech's fastCPPtests.

The test configuration is handled with CTest, so the test is only built if the appropriate CMake flag, ENABLE_TESTS, is ON.

The regression test itself is driven by reg_tests/executeRegressionTestCase.py which runs the OpenFAST test case and compares the binary output to the "gold standard" with reg_tests/pass_fail.py. The pass/fail criteria is a comparison of the variance of each channel in the binary output file with a predetermined tolerance. The tolerance is automatically picked by CMake based on machine and compiler type.

The test data is included in a submodule called RegressionTestData which based on the CertTest directory in FAST v8. This repository has two subdirectories: inputs and outputs. The contents of inputs are used in CTest configuration and the contents of outputs are the "gold standard" solutions. There is also a python script runCertTestsLocally.py which executes all of the cases in inputs in a new directory outputs-local.

To run the test from a fresh clone:
git clone https://github.com/rafaelmudafort/openfast.git
cd openfast
git checkout dev
git submodule update --init --recursive
mkdir build
cd build
ccmake .. # or otherwise configure cmake flags
-- hit c, then e
-- turn on the test boolean, configure the install prefix, etc
-- hit c, then e
-- again hit c, then e
-- hit g
make
make test

Improvements to come:

  • determine appropriate machine/compiler specific tolerances
  • add support in ctest for selecting a target reference directory
    • this would allow a developer to generate their own "gold standards" in case of new feature implementation or improvement
  • cron job & CDash

rafmudaf added 30 commits May 1, 2017 11:36
@ghost ghost closed this May 26, 2017
@ghost ghost mentioned this pull request Apr 16, 2019
@caroledaniel caroledaniel mentioned this pull request Feb 27, 2020
@HYFOAM HYFOAM mentioned this pull request Nov 29, 2020
@Tommi1198 Tommi1198 mentioned this pull request Apr 13, 2022
psakievich pushed a commit to psakievich/openfast that referenced this pull request Aug 3, 2023
andrew-platt pushed a commit that referenced this pull request Mar 11, 2024
@iosaaaa iosaaaa mentioned this pull request Aug 19, 2024
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants