Change test cases to run like suites

[xylar]

This merge makes several changes that are useful in preparing for task parallelism.

The most noticeable is that it gets rid of run_test_case() and instead uses run_tests() [renamed from run_suite()] to run both suites and single test cases. This has at least 2 potential advantages: it reduces redundant code and it gives additional useful output when running a test case on its own (notably the timing of the test case).

The other change will not be noticed by users. I moved compass/run.py into compass/run/serial.py in anticipation of having a second compass/run/???.py module that will handle task-parallel execution.

The merge also updates the documentation to account for the new locations of the functions and the new behavior.

[xylar]

Example output

Running the ocean/global_convergence/cosine_bell test case now produces:

ocean/global_convergence/cosine_bell
compass calling: compass.ocean.tests.global_convergence.cosine_bell.CosineBell.run()
  in /gpfs/fs1/home/ac.xylar/mpas-work/compass/run_test_cases_like_suites/compass/ocean/tests/global_convergence/cosine_bell/__init__.py

  * Cached step: QU60_mesh
  * Cached step: QU60_init
  * step: QU60_forward
  * Cached step: QU90_mesh
  * Cached step: QU90_init
  * step: QU90_forward
  * Cached step: QU120_mesh
  * Cached step: QU120_init
  * step: QU120_forward
  * Cached step: QU150_mesh
  * Cached step: QU150_init
  * step: QU150_forward
  * Cached step: QU180_mesh
  * Cached step: QU180_init
  * step: QU180_forward
  * Cached step: QU210_mesh
  * Cached step: QU210_init
  * step: QU210_forward
  * Cached step: QU240_mesh
  * Cached step: QU240_init
  * step: QU240_forward
  * step: analysis

compass calling: compass.ocean.tests.global_convergence.cosine_bell.CosineBell.validate()
  inherited from: compass.testcase.TestCase.validate()
  in /gpfs/fs1/home/ac.xylar/mpas-work/compass/run_test_cases_like_suites/compass/testcase.py

  test execution:      SUCCESS
Test Runtimes:
03:32 PASS ocean_global_convergence_cosine_bell
Total runtime 03:32
PASS: All passed successfully!

[xylar]

Testing

I ran the ocean pr test suite on Anvil with Intel and Intel-MPI, using a baseline from yesterday. All tests passed and the suite output looked the same as usual.

I also ran several test cases and steps. Step output looked the same as usual. Test case output looked like the example above.

[xylar]

@mark-petersen, @matthewhoffman and @altheaden, this is ready for your comments and review.

This work is needed in order for @altheaden to make headway so it is not super urgent but would be helpful to have in the next 2-3 weeks at the latest.

@mark-petersen and @matthewhoffman, hopefully, it's easy to review. What I would be looking for (assuming you're okay with the concept) is to set up the branch and run a test case on its own to make sure 1) it works and 2) you're happy with the output.

[matthewhoffman]

@xylar & @altheaden , I'm starting to look at this now and hope to have completed my testing tomorrow. After perusing this initially, the only thing I noticed that is worth discussing so far is the naming convention - I like the concept of merging the two run functions, but I'm wondering if the name for that combined function deserves further thought. Using run_suite for both suites and test cases seems a little confusing. If anything calling it run_tests/run_test seems a more appropriately general name. (I also considered suggesting run_compass but I think that could be confusing for users who might also think of the setup phase as "running" compass.) What do you think about the name?

[xylar]

I'm fine with run_tests.

[xylar]

I rebuilt the docs and ran a local suite, test case and step to make sure they all work with the new run_tests name. Seems fine.

[matthewhoffman]

@xylar , I ran a suite before and after this change. I looked at the output for a single test that was part of the suite for both versions, and it looked identical. Is that expected? (I would guess yes, but just confirming I'm not missing something.)

[matthewhoffman]

@xylar, I ran a test case manually before and after the change. The image below shows the diff in the information printed to the terminal before (left) and after (right). It looks like a convenient side effect for running test cases like suites is that when a test case is run, you get a little more information printed: the name of the test at the beginning and a summary of PASS/FAIL and runtimes. That's a nice addition. The only thing we've lost is the Running steps: setup_mesh, 1proc_run, 4proc_run line before the steps are run. That would be kind of nice to have around, but not very important to me. I.e., if it's trivial to add that back, it would be kind of nice, but otherwise don't bother. (I wouldn't have noticed the difference if I wasn't looking at them side by side.)

[matthewhoffman]

@xylar , I'm approving this with 1 minor typo to fix in the docs and pending your judgment of the two general comments I made. I can see how this change will be really advantageous in the long run.

[xylar]

@matthewhoffman, thank you for the review. I addressed your comments, including adding back text like:

Running steps: setup_mesh, 1proc_run, 4proc_run

[matthewhoffman]

Thanks, @xylar ! And thanks for adding back that extra line of output.

[matthewhoffman]

@mark-petersen , I tested this pretty thoroughly, so it's probably sufficient for you to just ensure a test case runs fine for you.

[altheaden]

That all looks good to me, and I agree about the run_tests/run_test change.

[mark-petersen]

Tested on grizzly with gnu and nightly suite, also single test case. Everything works great, thanks.

[xylar]

Thanks, @mark-petersen!