add tdir icepack.setup option to define test directory (#253)

doc/source/user_guide/ug_running.rst

@@ -36,7 +36,7 @@ There are three usage modes,
 
 All modes will require use of ``--mach`` or ``-m`` to specify the machine and case and test modes 
 can use ``--set`` or ``-s`` to define specific options.  ``--test`` and ``--suite`` will require ``--testid`` to be set 
-and both of the test modes can use ``--bdir``, ``--bgen``, ``--bcmp``, and ``--diff`` to generate (save) results and compare results with prior results.
+and both of the test modes can use ``--bdir``, ``--bgen``, ``--bcmp``, and ``--diff`` to generate (save) results and compare results with prior results as well as ``--tdir`` to specify the location of the test directory.
 Testing will be described in greater detail in the :ref:`testing` section.
 
 Again, ``icepack.setup --help`` will show the latest usage information including 

doc/source/user_guide/ug_testing.rst

@@ -45,6 +45,9 @@ For individual tests, the following command line options can be set
 ``--testid`` ID
      specifies the testid.  This is required for every use of ``--test`` and ``--suite``.  This is a user defined string that will allow each test to have a unique case and run directory name.  This is also required.
 
+``--tdir`` PATH
+     specifies the test directory.  Testcases will be created in this directory.  (default is .)
+
 ``--mach`` MACHINE (see :ref:`case_options`)
 
 ``--env`` ENVIRONMENT1 (see :ref:`case_options`)
@@ -224,10 +227,17 @@ individual cases and tests do::
   ./icepack.setup --suite base_suite --mach wolf --env gnu --testid myid
 
 Like an individual test, the ``--testid`` option must be specified and can be any 
-string.  Once the tests are complete, results can be checked by running the
+string.  
+
+If using the ``--tdir`` option, that directory must not exist before the script is run.  The tdir directory will be
+created by the script and it will be populated by all tests as well as scripts that support the test suite::
+
+  ./icepack.setup --suite base_suite --mach wolf --env gnu --testid myid --tdir /scratch/$user/testsuite.myid
+
+Once the tests are complete, results can be checked by running the
 results.csh script in the [suite_name].[testid]::
 
-  cd base_suite.[testid]
+  cd testsuite.[testid]
   ./results.csh
 
 The predefined test suites are defined under **configuration/scripts/tests** and 
@@ -276,6 +286,9 @@ following options are valid for suites,
 ``--acct`` ACCOUNT
   optional
 
+``--tdir`` PATH
+  optional
+
 ``--testid`` ID
   required
 
@@ -303,18 +316,28 @@ Test Suite Examples
     ::
 
       ./icepack.setup --suite base_suite --mach conrad --env cray --testid v01a
-      cd base_suite.v01a
+      cd testsuite.v01a
+      #wait for runs to complete
+      ./results.csh
+
+ 2) **Basic test suite with user defined test directory**
+
+    Specify suite, mach, env, testid.
+    ::
+
+      ./icepack.setup --suite base_suite --mach conrad --env cray --testid v01a --tdir /scratch/$user/ts.v01a
+      cd /scratch/$user/ts.v01a
       #wait for runs to complete
       ./results.csh
 
 
- 2) **Multiple test suites on multiple environments**
+ 3) **Multiple test suites on multiple environments**
 
       Specify multiple envs.
       ::
 
         ./icepack.setup --suite base_suite,quick_suite --mach conrad --env cray,pgi,intel,gnu --testid v01a
-        cd base_suite.v01a
+        cd testsuite.v01a
         #wait for runs to complete
         ./results.csh
 
@@ -323,52 +346,52 @@ Test Suite Examples
       of `icepack.setup` but if that approach is taken, it is recommended that different testids be used.
 
 
- 3) **Basic test suite, store baselines in user defined name**
+ 4) **Basic test suite, store baselines in user defined name**
 
       Add ``--bgen``
       ::
 
         ./icepack.setup --suite base_suite --mach conrad --env cray --testid v01a --bgen icepack.v01a
-        cd base_suite.v01a
+        cd testsuite.v01a
         #wait for runs to complete
         ./results.csh
 
       This will store the results in the default [bdir] directory under the subdirectory icepack.v01a.
 
 
- 4) **Basic test suite, store baselines in user defined top level directory**
+ 5) **Basic test suite, store baselines in user defined top level directory**
 
       Add ``--bgen`` and ``--bdir``
       ::
 
         ./icepack.setup --suite base_suite --mach conrad --env cray --testid v01a --bgen icepack.v01a --bdir /tmp/user/ICEPACK_BASELINES
-        cd base_suite.v01a
+        cd testsuite.v01a
         #wait for runs to complete
         ./results.csh
 
       This will store the results in /tmp/user/ICEPACK_BASELINES/icepack.v01a.
 
 
- 5) **Basic test suite, store baselines in auto-generated directory**
+ 6) **Basic test suite, store baselines in auto-generated directory**
 
       Add ``--bgen default``
       ::
 
         ./icepack.setup --suite base_suite --mach conrad --env cray --testid v01a --bgen default
-        cd base_suite.v01a
+        cd testsuite.v01a
         #wait for runs to complete
         ./results.csh
 
       This will store the results in the default [bdir] directory under a directory name generated by the script that includes the hash and date.
 
 
- 6) **Basic test suite, compare to prior baselines**
+ 7) **Basic test suite, compare to prior baselines**
 
       Add ``--bcmp``
       ::
 
         ./icepack.setup --suite base_suite --mach conrad --env cray --testid v02a --bcmp icepack.v01a
-        cd base_suite.v02a
+        cd testsuite.v02a
         #wait for runs to complete
         ./results.csh
 
@@ -377,7 +400,7 @@ Test Suite Examples
       (``--bdir`` and ``--bgen``)
 
 
- 7) **Basic test suite, use of default string in regression testing**
+ 8) **Basic test suite, use of default string in regression testing**
 
       default is a special argument to ``--bgen`` and ``--bcmp``.  When used, the
       scripts will automate generation of the directories.  In the case of ``--bgen``,
@@ -405,7 +428,7 @@ Test Suite Examples
       results each time without having to change the arguments.
 
 
- 8) **Create and test a custom suite**
+ 9) **Create and test a custom suite**
 
       Create your own input text file consisting of 5 columns of data,
        - Test
@@ -453,7 +476,7 @@ To post results, once a test suite is complete, run ``results.csh`` and
 ::
 
   ./icepack.setup --suite base_suite --mach conrad --env cray --testid v01a
-  cd base_suite.v01a
+  cd testsuite.v01a
   #wait for runs to complete
   ./results.csh
   ./report_results.csh
@@ -490,7 +513,7 @@ $ ./icepack.setup -m conrad -e intel --suite base_suite -acct <account_number> -
 
 Wait for suite to finish then go to the directory. ::
 
-$ cd base_suite.t00
+$ cd testsuite.t00
 
 Run the timeseries script on the desired case. ::
 

icepack.setup

@@ -19,6 +19,7 @@ set test = ${spval}
 set grid = col
 set pesx = 1x1
 set sets = ""
+set tdir = ${spval}
 set bdir = ${spval}
 set testid = ${spval}
 set testsuite = ${spval}
@@ -73,10 +74,11 @@ SYNOPSIS
     --test TEST -m MACH --testid ID 
         [-e ENV][-p MxN][-g GRID][-s SET1,SET2][--acct ACCT]
         [--diff TESTNAME][--bdir DIR][--bgen DIR][--bcmp DIR]
+        [--tdir PATH]
 
     --suite SUITE[,SUITE2] -m MACH --testid ID 
         [-e ENV1,ENV2][--acct ACCT][--bdir DIR][--bgen DIR]
-        [--bcmp DIR][--report]
+        [--bcmp DIR][--tdir PATH][--report]
 
 DESCRIPTION
     --help, -h : help
@@ -95,9 +97,10 @@ DESCRIPTION
 
     --test     : test, test name (not with --case or --suite)
     --suite    : test suite, pre-defined set or sets of tests, comma separated (not with --case or --test)
-    --bdir     : top baseline directory, default ICE_MACHINE_BASELINE
-    --bgen     : baselines directory where output from current tests are copied
-    --bcmp     : baselines directory where output from current tests are compared
+    --tdir     : directory name where tests will be located
+    --bdir     : baseline directory for regression testing, default ICE_MACHINE_BASELINE
+    --bgen     : directory name where output from current tests are copied
+    --bcmp     : directory name where output from current tests are compared
     --testid   : test ID, user-defined id for testing (REQUIRED with --test or --suite)
     --diff     : generate comparison against another case
     --report   : automatically post results when tests are complete
@@ -232,6 +235,8 @@ while (1)
     else if ("$option" == "--test") then
       set test = $argv[1]
       set dotest = 1
+    else if ("$option" == "--tdir") then
+      set tdir = $argv[1]
     else if ("$option" == "--grid" || "$option" == "-g") then
       set grid = $argv[1]``
     else if ("$option" == "--queue") then
@@ -312,6 +317,9 @@ if ( ${dosuite} == 1 ) then
   set tsdir  = "testsuite.${testid}"
   set tsfile = "testsuite.${testid}.${sdate}-${stime}.list"
 endif
+if ( ${tdir} != ${spval} ) then
+  set tsdir = ${tdir}
+endif
 if (-e $tsfile) then
   echo "${0}: ERROR in tsfile, this should never happen"
   exit -1
@@ -346,20 +354,24 @@ else
     endif
   end
 
-  if (-e ./${tsdir}) then
+  if (-e ${tsdir}) then
     echo "${0}: ERROR, ${tsdir} already exists"
     exit -1
   endif
 
-  mkdir ./${tsdir}
-  cp -f ${ICE_SCRIPTS}/tests/report_results.csh ./${tsdir}
-  cp -f ${ICE_SCRIPTS}/tests/timeseries.csh ./${tsdir}
+  mkdir -p ${tsdir}
+  if ($status != 0) then
+    echo "${0}: ERROR, mkdir ${tsdir} aborted"
+    exit -1
+  endif
+  cp -f ${ICE_SCRIPTS}/tests/report_results.csh ${tsdir}
+  cp -f ${ICE_SCRIPTS}/tests/timeseries.csh ${tsdir}
 
   if ($report == 1) then
-    cp -f ${ICE_SCRIPTS}/tests/poll_queue.csh ./${tsdir}
+    cp -f ${ICE_SCRIPTS}/tests/poll_queue.csh ${tsdir}
   endif
 
-cat >! ./${tsdir}/suite.run << EOF0
+cat >! ${tsdir}/suite.run << EOF0
 #!/bin/csh -f
 EOF0
 
@@ -373,7 +385,7 @@ EOF0
   set vers = ${ICE_VERSION}
   set shhash = `echo ${hash} | cut -c 1-10`
 
-cat >! ./${tsdir}/results.csh << EOF0
+cat >! ${tsdir}/results.csh << EOF0
 #!/bin/csh -f
 rm -f results.log
 echo "#------- " >> results.log
@@ -391,8 +403,8 @@ echo "#vers = ${vers}" >> results.log
 echo "#------- " >> results.log
 EOF0
 
-  chmod +x ./${tsdir}/suite.run
-  chmod +x ./${tsdir}/results.csh
+  chmod +x ${tsdir}/suite.run
+  chmod +x ${tsdir}/results.csh
 
 endif
 
@@ -466,9 +478,9 @@ foreach compiler ( $ncompilers )
       set testname_noid = "${machcomp}_${test}_${grid}_${pesx}${soptions}"
       set testname_base = "${machcomp}_${test}_${grid}_${pesx}${soptions}.${testid}"
       if (${dosuite} == 1) then
-        set testname = "${tsdir}/$testname_base"
+        set testname = "${tsdir}/${testname_base}"
       else
-        set testname = "$testname_base"
+        set testname = "${testname_base}"
       endif
       set case = ${testname}
     endif
@@ -771,12 +783,12 @@ EOF2
       cd ${ICE_SANDBOX}
       # Write build and run commands to suite.run
 
-cat >> ./${tsdir}/results.csh << EOF
-cat $testname_base/test_output >> results.log
+cat >> ${tsdir}/results.csh << EOF
+cat ${testname_base}/test_output >> results.log
 EOF
 
-cat >> ./${tsdir}/suite.run << EOF
-cd $testname_base
+cat >> ${tsdir}/suite.run << EOF
+cd ${testname_base}
 ./icepack.build
 ./icepack.submit
 cd ..
@@ -801,7 +813,7 @@ end
 if ( ${dosuite} == 1 ) then
 
   # Add code to results.csh to count the number of failures
-cat >> ./${tsdir}/results.csh << EOF
+cat >> ${tsdir}/results.csh << EOF
 cat ./results.log
 set pends = \`cat ./results.log | grep PEND | wc -l\`
 set failures = \`cat ./results.log | grep FAIL | wc -l\`