Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move check result to target repo (bot/check-result.sh script) #174

Merged
merged 45 commits into from
Jun 20, 2023
Merged
Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
bb851f4
Merge commit 'refs/pull/171/head' of gh-nessi:EESSI/eessi-bot-softwar…
truib Apr 7, 2023
7bc0d23
fixing one test due to changed function definition
truib Apr 7, 2023
bb0e627
add base attribute to MockPullRequest
truib Apr 7, 2023
2e4c8d2
adding nested data structure used to get repository name
truib Apr 7, 2023
5a8628a
change attribute name for MockPullRequest
truib Apr 7, 2023
f441fb7
fix more failing tests due to changed function definitions
truib Apr 7, 2023
031ae88
fix flake8 issues
truib Apr 7, 2023
467bc5b
add new settings to app.cfg.example
truib Apr 7, 2023
21c79b8
move job metadata functions to separate module
truib Apr 8, 2023
b009625
add step to check result in job script
truib Apr 8, 2023
c8ef07a
move metadata function to separate module
truib Apr 8, 2023
936fef4
move metadata function to own module + sketch procedure to check job …
truib Apr 8, 2023
526626d
make read metadata function more generic
truib Apr 8, 2023
50d9f4b
function to read job result + change in result file format
truib Apr 8, 2023
ece09ed
first sketch of using job result
truib Apr 8, 2023
141c719
introduce namedtuple PRComment + use it for create_metadata_file + tests
truib Apr 9, 2023
487ddde
format details as list items + update comment + polishing
truib Apr 9, 2023
be5445b
restructure code + provide results template
truib Apr 9, 2023
367f248
address flake8 issues
truib Apr 9, 2023
2073965
tweaking messages generated by the bot's job script
truib Apr 10, 2023
6127332
do not include empty lines in job status lists
truib Apr 10, 2023
a29d950
fixing bugs + tweaking some messages
truib Apr 10, 2023
517322e
improve job result comment template
truib Apr 10, 2023
8beab19
removed leftover from testing
truib Apr 10, 2023
b5b765e
need to remove non bot jobs earlier + using current jobs for processi…
truib Apr 14, 2023
ef947e4
improvements for using results produced by bot/check-result.sh script
truib Apr 19, 2023
05afc40
updating format, adding missing settings
truib Apr 19, 2023
52f1455
polishing of job result details
truib Apr 20, 2023
e9f9ae0
let bot/check-result.sh provide fully defined comment details
truib Apr 27, 2023
ee3a631
removed all software-layer specific code that checked result of a bui…
truib Apr 27, 2023
ace99ec
renamed comment_details to comment_description + clarification of res…
truib Apr 28, 2023
15cded1
Merge branch 'main' of gh-nessi:EESSI/eessi-bot-software-layer into m…
truib May 16, 2023
2ed8f78
Merge branch 'main' of https://github.com/EESSI/eessi-bot-software-la…
truib Jun 17, 2023
03d9472
remove unused import
truib Jun 17, 2023
c43c04a
fix arg name
truib Jun 18, 2023
b67d17d
update search pattern for No missing ...
truib Jun 18, 2023
e67f7e8
add more logging + fix search pattern
truib Jun 18, 2023
c1f3d3d
fix pattern to check for missing installations
truib Jun 18, 2023
34021f2
trivial import fixes
boegel Jun 19, 2023
615d71a
Merge branch 'main' into move_check_result_to_target_repo
boegel Jun 19, 2023
a9f8cb2
fix test_create_metadata_file
boegel Jun 19, 2023
625da33
also test use of read_metadata_file function
boegel Jun 19, 2023
82ac4ae
fix fetching PR comment ID in submit_build_jobs
boegel Jun 19, 2023
c627f16
use `comment_description` rather than `comment_description` in `*.res…
boegel Jun 20, 2023
23b1e99
added missing settings to job result file created by bot-build.slurm
truib Jun 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 44 additions & 17 deletions app.cfg.example
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ private_key = PATH_TO_PRIVATE_KEY
# which GH accounts have the permission to send commands to the bot
# if value is left/empty everyone can send commands
# value can be a space delimited list of GH accounts
#
# NOTE, be careful to not list the bot's name (GH app name) or it may consider
# comments created/updated by itself as a bot command and thus enter an
# endless loop.
Expand Down Expand Up @@ -55,13 +56,13 @@ container_cachedir = PATH_TO_SHARED_DIRECTORY
# it may happen that we need to customize some CVMFS configuration
# the value of cvmfs_customizations is a dictionary which maps a file
# name to an entry that needs to be added to that file
cvmfs_customizations = { "/etc/cvmfs/default.local": "CVMFS_HTTP_PROXY=\"http://PROXY_DNS_NAME:3128|http://PROXY_IP_ADDRESS:3128\"" }
# cvmfs_customizations = { "/etc/cvmfs/default.local": "CVMFS_HTTP_PROXY=\"http://PROXY_DNS_NAME:3128|http://PROXY_IP_ADDRESS:3128\"" }

# if compute nodes have no internet connection, we need to set http(s)_proxy
# or commands such as pip3 cannot download software from package repositories
# for example, the temporary EasyBuild is installed via pip3 first
http_proxy = http://PROXY_DNS:3128/
https_proxy = http://PROXY_DNS:3128/
# http_proxy = http://PROXY_DNS:3128/
# https_proxy = http://PROXY_DNS:3128/

# directory under which the bot prepares directories per job
# structure created is as follows: YYYY.MM/pr_PR_NUMBER/event_EVENT_ID/run_RUN_NUMBER/OS+SUBDIR
Expand Down Expand Up @@ -95,9 +96,12 @@ submit_command = /usr/bin/sbatch
# the label 'bot:build' (apparently this cannot be restricted on GitHub)
# if value is left/empty everyone can trigger the build
# value can be a space delimited list of GH accounts
build_permission = Hafsa-Naeem
build_permission =

# template for comment when user who set a label has no permission to trigger build jobs
no_build_permission_comment = Label `bot:build` has been set by user `{build_labeler}`, but this person does not have permission to trigger builds


[deploycfg]
# script for uploading built software packages
tarball_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging
Expand Down Expand Up @@ -128,7 +132,9 @@ upload_policy = once
# the label 'bot:deploy' (apparently this cannot be restricted on GitHub)
# if value is left/empty everyone can trigger the deployment
# value can be a space delimited list of GH accounts
deploy_permission = trz42
deploy_permission =

# template for comment when user who set a label has no permission to trigger deploying tarballs
no_deploy_permission_comment = Label `bot:deploy` has been set by user `{deploy_labeler}`, but this person does not have permission to trigger deployments


Expand All @@ -138,6 +144,16 @@ no_deploy_permission_comment = Label `bot:deploy` has been set by user `{deploy_
arch_target_map = { "linux/x86_64/generic" : "--constraint shape=c4.2xlarge", "linux/x86_64/amd/zen2": "--constraint shape=c5a.2xlarge" }


[repo_targets]
# defines for which repository a arch_target should be build for
#
# EESSI/2021.12 and NESSI/2022.11
repo_target_map = { "linux/x86_64/amd/zen2" : ["eessi-2021.12","nessi.no-2022.11"] }

# points to definition of repositories (default EESSI-pilot defined by build container)
repos_cfg_dir = PATH_TO_SHARED_DIRECTORY/cfg_bundles


# configuration for event handler which receives events from a GitHub repository.
[event_handler]
# path to the log file to log messages for event handler
Expand All @@ -161,23 +177,34 @@ poll_interval = 60
# full path to the command for manipulating existing jobs
scontrol_command = /usr/bin/scontrol

# variable 'initial_comment' under 'submitted_job_comments' should not be changed as the bot's code uses regular expression pattern to identify a matching comment

# Note 1. The value of the setting 'initial_comment' in section
# '[submitted_job_comments]' should not be changed because the bot
# uses regular expression pattern to identify a comment with this
# format.
# Note 2. Any name inside curly brackets is replaced by the bot with
# corresponding data. If the name is changed or the curly brackets
# are removed, the output (in PR comments) will lack important
# information.
[submitted_job_comments]
initial_comment = New job on instance `{app_name}` for architecture `{arch_name}` for repository `{repo_id}` in job dir `{symlink}`
awaits_release = job id `{job_id}` awaits release by job manager
initial_comment = New job on instance `{app_name}` for architecture `{arch_name}` for repository `{repo_id}` in job dir `{symlink}`
awaits_release = job id `{job_id}` awaits release by job manager


[new_job_comments]
awaits_launch = job awaits launch by Slurm scheduler

[running_job_comments]
running_job = job `{job_id}` is running
running_job = job `{job_id}` is running


[finished_job_comments]
success = :grin: SUCCESS tarball `{tarball_name}` ({tarball_size} GiB) in job dir
failure = :cry: FAILURE
no_slurm_out = No slurm output `{slurm_out}` in job dir
slurm_out = Found slurm output `{slurm_out}` in job dir
missing_modules = Slurm output lacks message "No missing modules!".
no_tarball_message = Slurm output lacks message about created tarball.
no_matching_tarball = No tarball matching `{tarball_pattern}` found in job dir.
multiple_tarballs = Found {num_tarballs} tarballs in job dir - only 1 matching `{tarball_pattern}` expected.
success = :grin: SUCCESS tarball `{tarball_name}` ({tarball_size} GiB) in job dir
failure = :cry: FAILURE
no_slurm_out = No slurm output `{slurm_out}` in job dir
slurm_out = Found slurm output `{slurm_out}` in job dir
missing_modules = Slurm output lacks message "No missing modules!".
no_tarball_message = Slurm output lacks message about created tarball.
no_matching_tarball = No tarball matching `{tarball_pattern}` found in job dir.
multiple_tarballs = Found {num_tarballs} tarballs in job dir - only 1 matching `{tarball_pattern}` expected.
job_result_unknown_fmt = <details><summary>:shrug: UNKNOWN _(click triangle for detailed information)_<summary/><ul><li>Job results file `{filename}` does not exist in job directory or reading it failed.</li><li>No artefacts were found/reported.</li></ul></details>
Loading