Add pointing interpolated values #223

morcuended · 2019-11-21T11:01:32Z

Here is the modification to add non-empty gps_time and interpolate the pointing values event-wise from the values contained in the Drive log files.

Something weird happened: I thought I had made a fresh installation of the lstchain repo yesterday, following the instructions written en the README. However, apparently the ctapipe-io-lst package was not pointing to the last version v0.2 as it is specified in environment.yml. So some names within the LSTDriveContainer were not properly set. I had to reinstall it manually to make everything work.

Disclaimer: When adding the pointing interpolated values, the code becomes substantially slower. Maybe this could be optimized. Perhaps it is due to the large file size of the Drive log.

Please have a look and let me know if you have any comment/suggestion

- Added filling gps_time field with the timestamps taken from UCTS - When the UCTS timestamps are not availables take time info from TIB NTP counters - This time will be used to interpolate the pointing direction for each event

- Takes the timing info previously defined

- The drive log with pointing can be parse as an input optional argument. - The pointing info is added only when this file is parsed. - Otherwise the only new parameter that dl1 files contain is gps_time taken from UCTS or TIB.

lstchain/reco/dl0_to_dl1.py

morcuended · 2019-11-21T13:12:54Z

This is an example of how to run the script adding pointing information:

python lstchain_data_r0_to_dl1.py -f ~/LST-1.1.Run01388.0000.fits.fz \
    -conf config.json \
    -pedestal ~/pedestal_run1575_0000.fits \
    -calib ~/calibration_run1572.hdf5 \
    -pointing ~/drive_log/drive_log_19_09_19.txt

simply parsing the path to the drive log file. The problem at this moment is that one has to check first which drive report file should use because they usually mix several days.

Output:

vuillaut

Looks overall good.
I haven't tested with real data.
I am working on getting a real data sample for unit tests, in the meantime, did you run it on real data and get a proper DL1 output ?

vuillaut · 2019-11-21T14:17:34Z

lstchain/reco/dl0_to_dl1.py

+                        ucts_timestamp = event.lst.tel[telescope_id].evt.ucts_timestamp
+
+                        if ucts_timestamp != 0:
+                            ns = 1e-9  # Convert nanosecs to secs


Using astropy units has been on the todo list for a long time.
Starting now would be great :)

vuillaut · 2019-11-21T14:27:44Z

lstchain/reco/dl0_to_dl1.py

+                        dl1_container.gps_time = gps_time
+
+                        if pointing_file_path:
+                            pointing_file = pointing_file_path


morcuended · 2019-11-21T14:39:34Z

Looks overall good.
I haven't tested with real data.
I am working on getting a real data sample for unit tests, in the meantime, did you run it on real data and get a proper DL1 output ?

Yes, I tested it with one subrun of real data on my laptop I think getting proper output values.

vuillaut · 2019-11-21T16:26:56Z

@morcuended dl1ab test is failing.
can you see on your laptop why?

morcuended · 2019-11-21T16:37:23Z

@morcuended dl1ab test is failing.
can you see on your laptop why?

I think it is because that branch was merged in the meanwhile and I did not pull it to my local repo. Will fix it and push again

into pointing

morcuended · 2019-11-21T16:49:32Z

I do not understand why is failing though. The scripts I modified are not dependant on lstchain_mc_dl1ab.py. I pulled the changes on my local repo and ran pytest. It's failing too.

What should I do?

rlopezcoto · 2019-11-21T16:59:34Z

After pulling the master, did you merge it?
There should be some message of "merged master branch" after you push it and I do not see it.

morcuended · 2019-11-21T17:05:58Z

After pulling the master, did you merge it?
There should be some message of "merged master branch" after you push it and I do not see it.

Oh, I forgot that last step. I just pushed my branch again.

morcuended · 2019-11-21T17:12:45Z

Still failing

rlopezcoto · 2019-11-21T17:22:08Z

Last changes are not merged yet. Can you do:
git fetch upstream
git merge upstream/master
git push

vuillaut · 2019-11-21T17:22:21Z

@morcuended weird it's not failing on my system, I'm investigating.

morcuended · 2019-11-21T17:26:36Z

(dev-lstchain) [dmorcuende@Toshiba cta-lstchain]$ git fetch upstream 
remote: Enumerating objects: 111, done.
remote: Counting objects: 100% (111/111), done.
remote: Compressing objects: 100% (45/45), done.
remote: Total 125 (delta 75), reused 96 (delta 65), pack-reused 14
Receiving objects: 100% (125/125), 20.56 KiB | 0 bytes/s, done.
Resolving deltas: 100% (77/77), completed with 12 local objects.
From https://github.com/cta-observatory/cta-lstchain
 * [new branch]      ctapipe_master -> upstream/ctapipe_master
 * [new branch]      master     -> upstream/master
 * [new branch]      production_scripts -> upstream/production_scripts
 * [new tag]         v0.1.0     -> v0.1.0

(dev-lstchain) [dmorcuende@Toshiba cta-lstchain]$ git merge upstream/master
Already up-to-date.

(dev-lstchain) [dmorcuende@Toshiba cta-lstchain]$ git push
Everything up-to-date

vuillaut · 2019-11-21T18:14:10Z

@morcuended actually, master is failing....

morcuended · 2019-11-21T18:26:52Z

@morcuended actually, master is failing....

I see. However, when PR #222 was merged, all tests had passed, right?

vuillaut · 2019-11-21T18:28:38Z

@morcuended actually, master is failing....

I see. However, when PR #222 was merged, all tests passed, right?

That's why I am a bit confused right now. Working on patch...

vuillaut · 2019-11-21T20:39:28Z

#224 should fix the issue.
You'll need to pull master again afterwards.

rlopezcoto · 2019-11-22T06:59:57Z

@morcuended please merge the new master, after #224 fix tests should pass.

morcuended · 2019-11-22T08:12:25Z

@morcuended please merge the new master, after #224 fix tests should pass.

Just did it. I think tests are OK now. Thanks a lot, @vuillaut and @rlopezcoto

morcuended added 4 commits November 20, 2019 20:00

Add timing info from UCTS and TIB

3448cbf

- Added filling gps_time field with the timestamps taken from UCTS - When the UCTS timestamps are not availables take time info from TIB NTP counters - This time will be used to interpolate the pointing direction for each event

Some cleanup

124489a

Add pointing interpolated values

e97be07

- Takes the timing info previously defined

Add pointing optional argument

70677ca

- The drive log with pointing can be parse as an input optional argument. - The pointing info is added only when this file is parsed. - Otherwise the only new parameter that dl1 files contain is gps_time taken from UCTS or TIB.

morcuended commented Nov 21, 2019

View reviewed changes

lstchain/reco/dl0_to_dl1.py Outdated Show resolved Hide resolved

Fixed indentation

5053589

vuillaut reviewed Nov 21, 2019

View reviewed changes

Add astropy nanosecs and remove redundancies

645f747

Merge branch 'master' of https://github.com/cta-observatory/cta-lstchain

8d70f9a

into pointing

rlopezcoto mentioned this pull request Nov 21, 2019

Production scripts onsite #221

Closed

2 tasks

rlopezcoto mentioned this pull request Nov 22, 2019

Scripts fix #224

Merged

Resolve conflict in lstchain_data_r0_to_dl1.py

3af00cb

rlopezcoto self-requested a review November 22, 2019 08:15

rlopezcoto approved these changes Nov 22, 2019

View reviewed changes

rlopezcoto merged commit e48d9b2 into cta-observatory:master Nov 22, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pointing interpolated values #223

Add pointing interpolated values #223

morcuended commented Nov 21, 2019

morcuended commented Nov 21, 2019

vuillaut left a comment

vuillaut Nov 21, 2019

morcuended Nov 21, 2019

vuillaut Nov 21, 2019

morcuended Nov 21, 2019

morcuended commented Nov 21, 2019

vuillaut commented Nov 21, 2019 •

edited

Loading

morcuended commented Nov 21, 2019

morcuended commented Nov 21, 2019 •

edited

Loading

rlopezcoto commented Nov 21, 2019

morcuended commented Nov 21, 2019

morcuended commented Nov 21, 2019

rlopezcoto commented Nov 21, 2019

vuillaut commented Nov 21, 2019

morcuended commented Nov 21, 2019

vuillaut commented Nov 21, 2019

morcuended commented Nov 21, 2019 •

edited

Loading

vuillaut commented Nov 21, 2019

vuillaut commented Nov 21, 2019

rlopezcoto commented Nov 22, 2019

morcuended commented Nov 22, 2019

Add pointing interpolated values #223

Add pointing interpolated values #223

Conversation

morcuended commented Nov 21, 2019

morcuended commented Nov 21, 2019

vuillaut left a comment

Choose a reason for hiding this comment

vuillaut Nov 21, 2019

Choose a reason for hiding this comment

morcuended Nov 21, 2019

Choose a reason for hiding this comment

vuillaut Nov 21, 2019

Choose a reason for hiding this comment

morcuended Nov 21, 2019

Choose a reason for hiding this comment

morcuended commented Nov 21, 2019

vuillaut commented Nov 21, 2019 • edited Loading

morcuended commented Nov 21, 2019

morcuended commented Nov 21, 2019 • edited Loading

rlopezcoto commented Nov 21, 2019

morcuended commented Nov 21, 2019

morcuended commented Nov 21, 2019

rlopezcoto commented Nov 21, 2019

vuillaut commented Nov 21, 2019

morcuended commented Nov 21, 2019

vuillaut commented Nov 21, 2019

morcuended commented Nov 21, 2019 • edited Loading

vuillaut commented Nov 21, 2019

vuillaut commented Nov 21, 2019

rlopezcoto commented Nov 22, 2019

morcuended commented Nov 22, 2019

vuillaut commented Nov 21, 2019 •

edited

Loading

morcuended commented Nov 21, 2019 •

edited

Loading

morcuended commented Nov 21, 2019 •

edited

Loading