DQM gives Exception: lfns not found on GRID error, doesn't fetch run

[mdpunch]

Describe the bug

Running python ../nectarchain/src//nectarchain/dqm/start_dqm.py -r 4332 $NECTARCAMDATA $NECTARCAMDATA gives an exception.

To Reproduce
Steps to reproduce the behavior:

1. Start the nectarchain dev conda environment (version checked to be __version__== '0.1.7.dev1+gf6ca6b2')
2. dirac-configure or dirac-proxy-init
3. Check access to Dirac is granted (e.g. do cta-prod-show-dataset Prod6_Paranal_Single-telescope_NSB1x_muon_North_20deg_R1 and check for sensible output)
4. Run python {your_path_to_nectarchain_dev}/src/nectarchain/dqm/start_dqm.py -r 4332 $NECTARCAMDATA $NECTARCAMDATA
   Output is

$ python ../nectarchain/src//nectarchain/dqm/start_dqm.py -r 4332 $NECTARCAMDATA $NECTARCAMDATA
Input file path: /scr/punch/CTA/NectarCAM_muons/Beamer_runs
Output path: /scr/punch/CTA/NectarCAM_muons/Beamer_runs
2024-01-23 18:24:15,809 nectarchain.data.management WARNING run 4332 is not present in /scr/punch/CTA/NectarCAM_muons/Beamer_runs
NoneType: None
2024-01-23 18:24:16,163 nectarchain.data.management ERROR lfns not found on GRID
NoneType: None
Traceback (most recent call last):
  File "/scr/punch/CTA/NectarCAM_muons/../nectarchain/src//nectarchain/dqm/start_dqm.py", line 56, in <module>
    _, filelist = dm.findrun(args.runnb)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/scr/punch/CTA/nectarchain/src/nectarchain/data/management.py", line 39, in findrun
    lfns = DataManagement.get_GRID_location(run_number)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scr/punch/CTA/nectarchain/src/nectarchain/data/management.py", line 147, in get_GRID_location
    raise e
Exception: lfns not found on GRID

Expected behavior
As described in the DQM docs, I expected it to fetch the run concerned.

Supporting information

Additional context

[jlenain]

That's not a bug, but a behavior we already discussed several times in our software meetings. In the current lack of a run data base, runs are searched for through the NectarCAM E-log, which is assumed to be filled correctly, with the run path on DIRAC. On the implementation:

• the credentials to the E-log are not included in the public repo. Did you provide them to the DataManagement class?
• the underlying implementation is known to have hiccups, depending on the used web browser (results may vary). We intend soon to keep this mechanism as fallback, but rely on the DIRAC API of the dfind COMDIRAC command. An alternative would be to manually download run files in $NECTARCAMDATA before calling nectarchain.

[mdpunch]

So, in that case it should be recategorized to from bug to documentation problem or suchlike, maybe?

I can propose changes to the README.md. Also, for giving credentials for the e-log to the DataManagement class, this is not described in the documentation (maybe also discussed in the software meetings?); at least there is nothing found when searching for "credentials". So that could be added also (without giving the credentials, of course).

For now, I'll just download the runs, sure...

[jlenain]

Fixed in #90 , do you agree, @mdpunch ?

[mdpunch]

Very good, thanks!

Is that automagically taken into account by the DQM script now, or does the change have to be propagated?

[jlenain]

Hi @mdpunch ,
Yes, all code using the DataManagement class, which is also the case of the DQM, will benefit from it.

[mdpunch]

Great, then I agree it's fixed!

[jlenain]

Then closing this, fixed in #90