Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DIRAC management #103

Merged
merged 3 commits into from
Feb 2, 2024
Merged

Conversation

jlenain
Copy link
Collaborator

@jlenain jlenain commented Feb 2, 2024

This PR attempts to handle more gracefully exceptions occurring when DIRAC is initialized. DIRAC client initialization is needed to properly initiate communication with DIRAC services, however, this initialization can fail, for instance for jobs running on the grid (!). Here is an example log of a failed job on DIRAC:

<<<<<<<<<< dqm_processor.sh Standard Output >>>>>>>>>>

Cleaning environment 
Cleaning environment Done

Running
singularity exec --home /home/execute/dir_2367987/DIRAC_Kh6rZ0pilot/19832486 oras://ghcr.io/cta-observatory/nectarchain:latest /opt/conda/envs/nectarchain/bin/python /opt/cta/nectarchain/src/nectarchain/dqm/start_dqm.py --r0 -r 4968 /home/execute/dir_2367987/DIRAC_Kh6rZ0pilot/19832486/runs /home/execute/dir_2367987/DIRAC_Kh6rZ0pilot/19832486/NectarCAM_DQM_Run4968
Input file path: /home/execute/dir_2367987/DIRAC_Kh6rZ0pilot/19832486/runs
Output path: /home/execute/dir_2367987/DIRAC_Kh6rZ0pilot/19832486/NectarCAM_DQM_Run4968
2024-02-01 16:56:30 UTC Framework ERROR: No proxy found
2024-02-01 16:56:30 UTC Framework ERROR: No proxy found
2024-02-01 16:56:30 UTC Framework ERROR: No proxy found
2024-02-01 16:56:30 UTC Framework ERROR: No proxy found
2024-02-01 16:56:30 UTC Framework ERROR: No proxy found
2024-02-01 16:56:30 UTC Framework ERROR: No proxy found
2024-02-01 16:56:30 UTC Framework ERROR: No proxy found
2024-02-01 16:56:30 UTC Framework ERROR: No proxy found
 <<<<<<<<<< dqm_processor.sh Standard Error >>>>>>>>>>

./singularity_wrapper.sh: line 8: dconfig: command not found
./singularity_wrapper.sh: line 9: dinit: command not found
INFO:    Downloading oras image
Traceback (most recent call last):
  File "/opt/cta/nectarchain/src/nectarchain/dqm/start_dqm.py", line 62, in <module>
    from nectarchain.data.management import DataManagement
  File "/opt/cta/nectarchain/src/nectarchain/data/__init__.py", line 2, in <module>
    from .management import *
  File "/opt/cta/nectarchain/src/nectarchain/data/management.py", line 5, in <module>
    DIRAC.initialize()
  File "/opt/conda/envs/nectarchain/lib/python3.11/site-packages/DIRAC/__init__.py", line 258, in initialize
    raise exceptions.DIRACInitError(f"Failed to contact the Configuration Server: {retVal['Message']}")
DIRAC.Core.Utilities.exceptions.DIRACInitError: Failed to contact the Configuration Server: No proxy found

This happens because a DIRAC proxy init can not happen on a worker during a job.

@jlenain jlenain self-assigned this Feb 2, 2024
@jlenain jlenain requested a review from dkerszberg February 2, 2024 09:51
Copy link

codecov bot commented Feb 2, 2024

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (a31bad2) 30.92% compared to head (d711069) 30.92%.

Files Patch % Lines
src/nectarchain/data/management.py 55.55% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #103      +/-   ##
==========================================
- Coverage   30.92%   30.92%   -0.01%     
==========================================
  Files          60       60              
  Lines        3754     3758       +4     
==========================================
+ Hits         1161     1162       +1     
- Misses       2593     2596       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jlenain jlenain removed the request for review from dkerszberg February 2, 2024 10:32
@jlenain jlenain merged commit 8cb6ff9 into cta-observatory:master Feb 2, 2024
11 of 12 checks passed
@jlenain jlenain deleted the dirac-management branch February 2, 2024 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant