Merge pull request #1 from prisms-center/4.X

4.X
prisms-center · Oct 17, 2017 · b6b1aae · b6b1aae
2 parents 31dea83 + df26998
commit b6b1aae
Show file tree

Hide file tree

Showing 28 changed files with 913 additions and 532 deletions.
diff --git a/README.md b/README.md
@@ -33,8 +33,8 @@ Possible values for "taskstatus" are:
 
 
 Jobs are marked 'auto' either by submitting through the python class ``prisms_jobs.Job`` 
-with the attribute ``auto=True``, or by submitting a PBS script which contains 
-the line ``#auto=True`` using the included ``psub`` script.  
+with the attribute ``auto=True``, or by submitting a script which contains 
+the line ``#auto=True`` using the included ``psub`` command line program.  
 
 Jobs can be monitored using the command line program ``pstat``. All 'auto' jobs 
 which have stopped can be resubmitted using ``pstat --continue``. In this case, 
@@ -70,7 +70,7 @@ Jobs not marked 'auto' are shown with the status "Check" in ``pstat`` until the
 marks them as "Complete".
 
 
-## Installation from PyPI (todo)
+## Installation from PyPI
 
 Using ``pip``:
 
@@ -85,7 +85,7 @@ If installing to a user directory, you may need to set your PATH to find the ins
    	export PATH=$PATH:`python -m site --user-base`/bin
 
 
-## Install using conda (todo)
+## Install using conda
 
     conda config --add channels prisms-center
     conda install prisms-jobs
@@ -99,9 +99,9 @@ If installing to a user directory, you may need to set your PATH to find the ins
         git clone https://github.com/prisms-center/prisms_jobs.git
         cd prisms_jobs
 
-2. Checkout the branch/tag containing the version you wish to install. Latest is ``v3.0.1``:
+2. Checkout the branch/tag containing the version you wish to install. Latest is ``v4.0.0``:
 
-        git checkout v3.0.1
+        git checkout v4.0.0
 
 2. From the root directory of the repository:
 

diff --git a/build_conda.sh b/build_conda.sh
@@ -1,6 +1,5 @@
 # begin
 anaconda login
-conda config --set anaconda_upload yes
 
 # build, get location of result, upload
 conda build conda-recipes/prisms_jobs > conda-recipes/tmp.out
@@ -19,5 +18,4 @@ conda convert --platform linux-64 $LOCATION -o conda-recipes
 anaconda upload --user prisms-center conda-recipes/linux-64/prisms-jobs*
 
 # finish
-conda config --set anaconda_upload no
 anaconda logout
diff --git a/conda-recipes/prisms_jobs/meta.yaml b/conda-recipes/prisms_jobs/meta.yaml
@@ -1,18 +1,22 @@
 package:
   name: prisms-jobs
-  version: "3.0.1"
+  version: "4.0.0"
 
 source:
-  git_rev: v3.0.1
+  git_rev: v4.0.0
   git_url: https://github.com/prisms-center/prisms_jobs.git
 
 requirements:
   build:
     - python
     - setuptools
+    - future
+    - six
     - argparse # [py26]
   run:
     - python
+    - future
+    - six
     - argparse # [py26]
 
 about:

diff --git a/doc/source/api/index.rst b/doc/source/api/index.rst
@@ -16,7 +16,6 @@ prisms_jobs
     prisms_jobs.EligibilityError
     prisms_jobs.complete_job
     prisms_jobs.error_job
-    prisms_jobs.set_software
 
 prisms_jobs.interface
 ---------------------
@@ -27,6 +26,26 @@ prisms_jobs.interface
     prisms_jobs.interface.torque
     prisms_jobs.interface.slurm
     prisms_jobs.interface.default
+
+prisms_jobs.config
+---------------------
+
+.. autosummary::
+    :toctree:
+
+    prisms_jobs.config.configure
+    prisms_jobs.config.dbpath
+    prisms_jobs.config.settings
+    prisms_jobs.config.read_config
+    prisms_jobs.config.write_config
+    prisms_jobs.config.default_settings
+    prisms_jobs.config.config_dir
+    prisms_jobs.config.config_path
+    prisms_jobs.config.update_selection_method
+    prisms_jobs.config.set_update_selection_method
+    prisms_jobs.config.software
+    prisms_jobs.config.set_software
+    prisms_jobs.config.detect_software
 
 prisms_jobs.misc
 ----------------

diff --git a/doc/source/conf.py b/doc/source/conf.py
@@ -84,9 +84,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = u'3.0'
+version = u'4.0'
 # The full version, including alpha/beta/rc tags.
-release = u'3.0b0'
+release = u'4.0.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/doc/source/config.rst b/doc/source/config.rst
@@ -3,28 +3,90 @@
 Configuration
 =============
 
-Environment variables (typically not necessary):
+Some configuration is possible:
 
-- ``PRISMS_JOBS_DB``: 
+- ``PRISMS_JOBS_DIR``: (optional, default=``$HOME/.prisms_jobs``) 
 
-    The SQLite jobs database is stored by default at ``$HOME/.prisms_jobs/jobs.db``. 
-    If ``PRISMS_JOBS_DB`` is set, then the jobs database is stored at 
-    ``$PRISMS_JOBS_DB/jobs.db``.
+    The jobs database is stored at ``$PBS_JOB_DIR/jobs.db``.
 
-- ``PRISMS_JOBS_SOFTWARE``: 
+- ``PRISMS_JOBS_DIR/config.json``: 
 
-    By default, ``prisms-jobs`` will attempt to automatically 
-    detect ``'torque'`` (by checking for the 'qstat' executable) or ``'slurm'`` (by 
-    checking for the 'sbatch' executable). The ``'default'`` module provides stubs to 
-    enable testing/use on systems with no job management software. If ``PRISMS_JOBS_SOFTWARE``
-    is set to any other value, it is treated as the name of a Python module containing 
-    a custom interface which ``prisms_jobs`` will attempt to import and use.
+    Automatically generated JSON configuration file storing settings:
+
+    - ``"dbpath"``: (str) 
+
+        The location of the SQLite jobs database.
+
+    - ``"software"``: (str) 
+
+        The job submission software interface to use. ``"torque"`` or ``"slurm"``
+        is automatically detected if present.
+
+        +-------------------+------------------------------------------------+
+        |"torque"           |TORQUE                                          |
+        +-------------------+------------------------------------------------+
+        |"slurm"            |Slurm                                           |
+        +-------------------+------------------------------------------------+
+        |"default" (or null)|Empty stub, does nothing                        |
+        +-------------------+------------------------------------------------+
+        |other              |The name of an existing findable python module  |
+        |                   |implementing an interface                       |
+        +-------------------+------------------------------------------------+
 
-- ``PRISMS_JOBS_UPDATE``: 
+    - ``"write_submit_script"``: (bool, optional, default=false) 
+
+        If ``true``, submit jobs by first writing a submit script file and then 
+        submitting it. Otherwise, by default, the job is submitted via the command 
+        line.
+
+    - ``"update_method"``: (str, optional, default="default")
+
+        Controls which jobs are updated when JobDB.update() is called.
+
+        +-------------------+------------------------------------------------+
+        |"default" (or null)| Select jobs with jobstatus != 'C'              |
+        +-------------------+------------------------------------------------+
+        |"check_hostname"   | Select jobs with jobstatus != 'C' and matching |
+        |                   | hostname. This is useful on compute clusters   |
+        |                   | where multiple machines with different queues  |
+        |                   | share the same ``PRISMS_JOBS_DIR``.            |
+        +-------------------+------------------------------------------------+
+
+    - ``"taskmaster_job_kwargs"``: (JSON object, optional)
+
+        Holds options for the `taskmaster`_ job. Defaults are:
+
+        +-----------+------------------------------------------------+
+        |'name'     | "taskmaster"                                   |
+        +-----------+------------------------------------------------+
+        |'account'  | "prismsprojectdebug_fluxoe"                    |
+        +-----------+------------------------------------------------+
+        |'nodes'    | "1"                                            |
+        +-----------+------------------------------------------------+
+        |'ppn'      | "1"                                            |
+        +-----------+------------------------------------------------+
+        |'walltime' | "1:00:00"                                      |
+        +-----------+------------------------------------------------+
+        |'pmem'     | "3800mb"                                       |
+        +-----------+------------------------------------------------+
+        |'qos'      | "flux"                                         |
+        +-----------+------------------------------------------------+
+        |'queue'    | "fluxoe"                                       |
+        +-----------+------------------------------------------------+
+        |'message'  | null                                           |
+        +-----------+------------------------------------------------+
+        |'email'    | null                                           |
+        +-----------+------------------------------------------------+
+        |'priority' | "-1000"                                        |
+        +-----------+------------------------------------------------+
+        |'command'  | "rm taskmaster.o*; rm taskmaster.e*\\n"        |
+        +-----------+------------------------------------------------+
+        |'auto'     | false                                          |
+        +-----------+------------------------------------------------+
+
+        Additionally, the ``'exetime'`` is set based on the ``--delay`` 
+        commandline argument and the commandline invocation used to launch 
+        ``taskmaster`` is appended to ``'command'``.
 
-    If unset or set to ``'default'``, the ``pstat`` script 
-    will attempt to update the status of all jobs that are not yet complete (``'C'``). 
-    For systems with multiple-clusters-same-home, this may be set to ``'check_hostname'`` 
-    and ``pstat`` will only attempt to update the status of jobs that are not yet 
-    complete (``'C'``) and have matching hostname, as determined by ``socket.gethostname()``.
+.. _taskmaster: scripts/taskmaster.html
 
diff --git a/doc/source/index.rst b/doc/source/index.rst
@@ -1,7 +1,4 @@
-.. prisms_jobs documentation master file, created by
-   sphinx-quickstart on Mon Sep 11 12:22:02 2017.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
+.. prisms_jobs
 
 Welcome to prisms-jobs's documentation!
 =======================================

diff --git a/doc/source/install.rst b/doc/source/install.rst
@@ -45,11 +45,11 @@ Install from source
     git clone https://github.com/prisms-center/prisms_jobs.git
     cd prisms_jobs
 
-2. Checkout the branch/tag containing the version you wish to install. Latest is ``v3.0.1``:
+2. Checkout the branch/tag containing the version you wish to install. Latest is ``v4.0.0``:
 
 ::
 
-    git checkout v3.0.1
+    git checkout v4.0.0
 
 3. From the root directory of the repository:
 

diff --git a/doc/source/overview.rst b/doc/source/overview.rst
@@ -32,8 +32,8 @@ Possible values for "taskstatus" are:
 
 
 Jobs are marked 'auto' either by submitting through the python class ``prisms_jobs.Job`` 
-with the attribute ``auto=True``, or by submitting a PBS script which contains 
-the line ``#auto=True`` using the included ``psub`` script.  
+with the attribute ``auto=True``, or by submitting a script which contains 
+the line ``#auto=True`` using the included ``psub`` command line program.  
 
 Jobs can be monitored using the command line program ``pstat``. All 'auto' jobs 
 which have stopped can be resubmitted using ``pstat --continue``. In this case, 
@@ -61,7 +61,9 @@ Example screen shot:
 
 Additionally, when scheduling periodic jobs is not allowed other ways, the 
 ``taskmaster`` script can fully automate this process. ``taskmaster`` executes 
-``pstat --continue`` and then resubmits itself to execute again periodically.
+``pstat --continue`` and then resubmits itself to execute again periodically. As 
+not all compute resources allow this behavior, remember check the policy prior 
+to using ``taskmaster`` on a new compute resource.
 
 A script marked 'auto' should check itself for completion and when reached execute 
 ``pstat --complete $JOBID --force`` in bash, or ``prisms_jobs.complete_job()`` 

diff --git a/doc/source/scripts/pstat.rst b/doc/source/scripts/pstat.rst
@@ -1,12 +1,10 @@
 .. scripts/pstat.rst
 
-pstat
-=====
-
-
-Summary
--------
+``pstat``
+=========
 
+Summary:
+--------
 
 ``pstat`` gives command line access to the jobs database. It can be used to:
 
@@ -28,12 +26,11 @@ Summary
 - Delete jobs from the database (and abort if currently running)
 
 
-Help documentation:
--------------------
+``--help`` documentation:
+-------------------------
 
 .. argparse::
-    :filename: scripts/pstat
+    :filename: prisms_jobs/scripts/pstat.py
     :func: make_parser
     :prog: pstat
-
-
+
diff --git a/doc/source/scripts/psub.rst b/doc/source/scripts/psub.rst
@@ -1,15 +1,18 @@
 .. scripts/psub.rst
 
-psub
-====
+``psub``
+========
+
+Summary:
+--------
 
 ``psub`` submits a job script and adds the job to the job database.
 
-Help documentation:
--------------------
+``--help`` documentation:
+-------------------------
 
 .. argparse::
-    :filename: scripts/psub
+    :filename: prisms_jobs/scripts/psub.py
     :func: parser
     :prog: psub
 

diff --git a/doc/source/scripts/taskmaster.rst b/doc/source/scripts/taskmaster.rst
@@ -1,26 +1,28 @@
 .. scripts/taskmaster.rst
 
-taskmaster
-==========
+``taskmaster``
+==============
 
-``taskmaster`` submits a job on the PRISMS flux debug queue that will repeatedly
-resubmit any ``Auto`` jobs in the job database that have completed but whose
-taskstatus is still ``'Incomplete'`` (perhaps because the jobs has hit the walltime
-before completing or failed to converge) and then resubmit itself with a delay
-before execution.
+Summary:
+--------
 
-To use on machines other than flux change the line containing 
+``taskmaster`` submits a job that will repeatedly resubmit any ``Auto`` jobs in 
+the job database that have completed but whose taskstatus is still ``'Incomplete'`` 
+(perhaps because the jobs has hit the walltime before completing or failed to 
+converge) and then resubmit itself with a delay before execution. As not all
+compute resources allow this behavior, remember check the policy prior to using
+``taskmaster`` on a new compute resource.
 
-::
+The job submission options can be customized by editing the ``prisms-jobs``
+`configuration file`_.
 
-    j = prisms_jobs.templates.PrismsDebugJob(...)
 
-Help documentation:
--------------------
+``--help`` documentation:
+-------------------------
 
 .. argparse::
-    :filename: scripts/taskmaster
+    :filename: prisms_jobs/scripts/taskmaster.py
     :func: parser
     :prog: taskmaster
-    
-    
+
+_`configuration file`: config.html