Use ctapipe-process tools for the combined analysis pipeline

[YoshikiOhtani]

Stemming from the discussion in the issue #44, at some point we may try to process data using ctapipe-process tools and use the standard ctapipe DL1 and DL2 data format for the combined analysis pipeline. I think it is not high priority now, but it would be good to start a discussion and make a to-do list.

I think what we can already try to do is to use the ctapipe.io.DataWriter instead of ctapipe.io.HDFTableWriter in the magic_calib_to_dl1.py script, as it is done in scripts/mars/magic_calibrated_to_dl1.py. And referring to the issue cta-observatory/cta-lstchain#972, eventually it would be good to create a custom ImageProcessor for MAGIC events, and a custom ProcessTool for processing MAGIC calibrated data (load _Y_ file with MAGICEventSource, clean and parametrize the images, and store the data with DataWriter). Then, once the LST ImageProcessor is developed in lstchain, we may simply import the module and try to create a ProcessTool for LST+MAGIC MCs, which will be used in lst1_magic_mc_dl0_to_dl1.py

Or, probably we would not need to create a custom tools if we could modify the default ctapipe ProcessTool to be able to load MAGIC calibrated data, and use the MARS-like image cleaning.

I may misunderstand something, so any ideas and comments are very welcome!

[aleberti]

Yes, in principle what you say is correct. Regarding the MAGIC cleaning, I want to open a issue/PR in ctapipe for that. Satoshi recently merged a PR with the time constrained cleaning, there is another one open implementing ~MaTaJu cleaning. Those two contain all the things needed to implement a MARSImageCleaner properly, on top of which also unsuitable pixels treatment should be added.

Anyway you can refer to https://github.com/cta-observatory/ctapipe_io_magic/blob/master/ctapipe_io_magic/tests/test_stage1.py and https://github.com/cta-observatory/ctapipe_io_magic/blob/master/example_stage1_config.json to see for example how stage1 (i.e. calculation of DL1 parameters) works with the MAGICEventSource. One can pass options via the config file or via command line (like for the process_run flag) to set parameters of different event sources. What I did in magic_calibrated_to_dl1.py actually does the same thing as this ProcessorTool (I literally took the code from different parts to save everything in the std DL1 format). So magic_calib_to_dl1.py at the end would be really similar to test_stage1.py, with a different Image Cleaner.

[jsitarek]

@YoshikiOhtani, I agree that this is what we should aim to. The problem is that the current magic+lst1 scripts are using different philosophy than ctapipe data model, not only the actual names of containers are different, but also the merging of different data - in ctapipe you have the data separated in different containers simulation, dl1, etc, divided either by event or by telescope and you can connect them with the table rows, while in magic+lst1 part all the parameters are put together into one table, and different telescopes are written in separate rows of the table. For example I wanted to save /simulation/event/subarray/shower to save some extra information about the shower, but I realized that it does not make sense in the current scheme, because I cannot connect those information to the one in 'parameters'

@aleberti how do you do the merging of information from M1 and M2 in the ctapipe-like scheme, such that information in /event/dl1 and /simulation/event/subarray/shower are synchronized ?

[aleberti]

@aleberti how do you do the merging of information from M1 and M2 in the ctapipe-like scheme, such that information in /event/dl1 and /simulation/event/subarray/shower are synchronized ?

do you mean just merging the information (i.e. what merge_hdf_file does) or also the matching of stereo events? If you mean the former, it is easy to do it with ctapipe-merge. I tried some months ago and it worked. Of course the input should be in the standard DL1 format, which is done in mars/magic_calibrated_to_dl1.py

[jsitarek]

I mean specifically storing of the new tables after the stereo matching,
So something like this , e.g. you have 4 events: 1: M1+M2, 2: M1, 3: M1+M2, 4: M2
so in M1 file you have 1, 2, 3 and in M2 you have 1,3, 4, so in /simulation/event/subarray/shower you have at position [1] event 2 in M1 and 3 in M2.
I guess you have some loop over the L3 trigger number, and rebuild the stereo files from scratch, i.e. once you get a stereo event you add it to /simulation/event/subarray/shower, /dl1/event/telescope/.... etc.

[aleberti]

A step in this direction: cta-observatory/ctapipe#2511 with which we can implement the MARS cleaning as a derivate of ImageCleaner. But this will be available only from the next ctapipe release.