Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use job scheduling information to allow more flexible job submission #77

Merged
merged 49 commits into from
Jan 31, 2024
Merged

Use job scheduling information to allow more flexible job submission #77

merged 49 commits into from
Jan 31, 2024

Commits on Dec 11, 2023

  1. initial commit figuring out the general shape of it

    @thomasmfish
    thomasmfish committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    94e36b8 View commit details
    Browse the repository at this point in the history

  2. a bit more progress

    @thomasmfish
    thomasmfish committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    6ecb041 View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2023

  1. fix typo

    @thomasmfish
    thomasmfish committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    1c827d2 View commit details
    Browse the repository at this point in the history

  2. timestamp_ok fix to work with JSI.start_time instead of a scheduler… 

    …-wide one
    @thomasmfish
    thomasmfish committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    5d80e1b View commit details
    Browse the repository at this point in the history

  3. Add a timestamp value to JSI separate from the start time

    @thomasmfish
    thomasmfish committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    1e23d5c View commit details
    Browse the repository at this point in the history

  4. JobController updates

    @thomasmfish
    thomasmfish committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    185bda0 View commit details
    Browse the repository at this point in the history

  5. Remove duplicate key word argument

    @thomasmfish
    thomasmfish committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    17e3687 View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2024

  1. Rename SchedulerModeInterface to JobSlicerInterface

    @thomasmfish
    thomasmfish committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    f032b8b View commit details
    Browse the repository at this point in the history

  2. Various fixes

    @thomasmfish
    thomasmfish committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    e60b232 View commit details
    Browse the repository at this point in the history

  3. wait_all_jobs: work correctly with per-job timeouts

    @thomasmfish
    thomasmfish committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    03a2523 View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2024

  1. Check job_scheduling_info.log_directory exists and make it if not

    @thomasmfish
    thomasmfish committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    144e758 View commit details
    Browse the repository at this point in the history

  2. Add a terminate after total wait time to avoid getting stuck waiting … 

    …for jobs stuck in a queue
    @thomasmfish
    thomasmfish committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    89ffe32 View commit details
    Browse the repository at this point in the history

  3. use datetime and timedelta for StatusInfo

    @thomasmfish
    thomasmfish committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    34b579d View commit details
    Browse the repository at this point in the history

  4. Set time_to_dispatch and wall_time with the right type

    @thomasmfish
    thomasmfish committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    f7caa18 View commit details
    Browse the repository at this point in the history

  5. _wait_for_jobs working (to some extent!)

    @thomasmfish
    thomasmfish committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    9fdc467 View commit details
    Browse the repository at this point in the history

  6. some progress with the job scheduler tests

    @thomasmfish
    thomasmfish committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    2226817 View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2024

  1. get_output_paths: get output paths correctly from JSIs

    @thomasmfish
    thomasmfish committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    c7bea74 View commit details
    Browse the repository at this point in the history

  2. wait_for_ended: use the correct lists of jobs

    @thomasmfish
    thomasmfish committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    5db4ce0 View commit details
    Browse the repository at this point in the history

  3. _report_job_info check actual output file, as set in JSI

    @thomasmfish
    thomasmfish committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    468c409 View commit details
    Browse the repository at this point in the history

  4. get_job_history_batch: batch_number=None now returns latest batch

    @thomasmfish
    thomasmfish committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    7776238 View commit details
    Browse the repository at this point in the history

  5. use JobSchedulingInformation.set_job_script_path

    @thomasmfish
    thomasmfish committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    1c1712d View commit details
    Browse the repository at this point in the history

  6. Add final check_jobscript_is_readable check before sending for subm… 

    …ission
    @thomasmfish
    thomasmfish committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    e01fe15 View commit details
    Browse the repository at this point in the history

  7. Use consistent phrasing for testing simplicity

    @thomasmfish
    thomasmfish committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    aa669c9 View commit details
    Browse the repository at this point in the history

  8. filter_killed_jobs: correct typing

    @thomasmfish
    thomasmfish committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    f37ec35 View commit details
    Browse the repository at this point in the history

  9. resubmit_jobs: allow filtering by job_id

    @thomasmfish
    thomasmfish committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    d92e36c View commit details
    Browse the repository at this point in the history

  10. progress wth tests

    @thomasmfish
    thomasmfish committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    6cfafe2 View commit details
    Browse the repository at this point in the history

Commits on Jan 15, 2024

  1. test_job_scheduler_runs: correct create_js args

    @thomasmfish
    thomasmfish committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    aca8851 View commit details
    Browse the repository at this point in the history

  2. resubmit_killed_jobs and get_job_history_batch: Fixes for tests

    @thomasmfish
    thomasmfish committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    95c7bb7 View commit details
    Browse the repository at this point in the history

  3. All tests working (resubmit tests adapted)

    @thomasmfish
    thomasmfish committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    0c99d49 View commit details
    Browse the repository at this point in the history

  4. Remove old comment

    @thomasmfish
    thomasmfish committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    7335a78 View commit details
    Browse the repository at this point in the history

  5. Remove old variable from logging

    @thomasmfish
    thomasmfish committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    e8478a1 View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2024

  1. Remove StatusInfo.output_path in favour of using JSI, use stdout pa… 

    …th to check state
    @thomasmfish
    thomasmfish committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    3dad523 View commit details
    Browse the repository at this point in the history

  2. fetch_and_update_state use regex to check tres_alloc_str

    @thomasmfish
    thomasmfish committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    e005595 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2024

  1. String doesn't need to be f-string

    @thomasmfish
    thomasmfish committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    5b62b2e View commit details
    Browse the repository at this point in the history

  2. Replace leftover slots reference with cpus 

    Reformat long lines
    @PeterC-DLS
    PeterC-DLS committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    be20610 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2024

  1. Set minimum Python version to 3.10, plus use Python 3.10+ style typing

    @thomasmfish @PeterC-DLS
    thomasmfish authored and PeterC-DLS committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    22b8718 View commit details
    Browse the repository at this point in the history

  2. resubmit_killed_jobs: fix args (missing =)

    @thomasmfish @PeterC-DLS
    thomasmfish authored and PeterC-DLS committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    9fb6912 View commit details
    Browse the repository at this point in the history

  3. Remove unused imports

    @thomasmfish @PeterC-DLS
    thomasmfish authored and PeterC-DLS committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    e02a6da View commit details
    Browse the repository at this point in the history

  4. add minimum versions to meta.yaml and environment.yaml

    @thomasmfish @PeterC-DLS
    thomasmfish authored and PeterC-DLS committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    4669c02 View commit details
    Browse the repository at this point in the history

  5. Refactor job slicers 

    Add back methods in program wrapper and tweak job slicer implementations to
fix job controller tests, and fix job scheduler tests
    @PeterC-DLS
    PeterC-DLS committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    5ad4711 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2024

  1. Update version number

    @PeterC-DLS
    PeterC-DLS committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    a31d6b5 View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2024

  1. Fix some bugs and type annotations from mypy 

    Add back script names for some job slicers
    @PeterC-DLS
    PeterC-DLS committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    0086c5c View commit details
    Browse the repository at this point in the history

  2. Fix bugs in memory argument handling and make token optional

    @PeterC-DLS
    PeterC-DLS committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    52e1b59 View commit details
    Browse the repository at this point in the history

  3. Fix logic in job scheduler waiting method 

    Fix wrong destination of output in pass thru wrapper, and more logging info
    @PeterC-DLS
    PeterC-DLS committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    1b59af3 View commit details
    Browse the repository at this point in the history

  4. Format imports and code

    @PeterC-DLS
    PeterC-DLS committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    b271520 View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2024

  1. Fix expected string in test 

    Add documentation on testing and rename test env var for slurm partition
    @PeterC-DLS
    PeterC-DLS committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    e7ee00e View commit details
    Browse the repository at this point in the history

  2. Fix up job ID check 

    Simplify output path in pass thru wrapper, remove odd bits
    @PeterC-DLS
    PeterC-DLS committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    5d9b98b View commit details
    Browse the repository at this point in the history

  3. get_job: error messages needed to be f-strings

    @thomasmfish
    thomasmfish committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    4e44dd5 View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2024

  1. Set fallback job script in job slicer to current working directory

    @PeterC-DLS
    PeterC-DLS committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    c9142bb View commit details
    Browse the repository at this point in the history