PyHESSIO module move_to_next_event not reading the correct event ID for triggered events

[FrancescoGSaturni]

When calling the move_to_next_event module in order to generate the triggered event stream from sim_telarray output produced with its last version (2018/11/07), the event ID associated to each event is apparently not read but set to a sequential value up to the number of triggers. In other words, instead of reading e.g. this:

EVTNUM
1 2401
2 11100
3 13006
4 21910
5 22316
6 23405
...

I am reading this:

EVTNUM
1 1
2 2
3 3
4 4
5 5
6 6
…

Apparently, the Python function is not reading any triggered event ID (therefore leaving the values of col. EVTNUM to their initializations). I have already verified that the datum is not missing from the sim_telarray output thanks to the ROOT converter EventDisplay. Note that this problem does not hold when trying instead to read MC events through move_to_next_event(event_type = 2021), which are recorded with the correct event ID (i.e. evt_num = 100*n + p, n being the shower number and p the telescope random position).

Note that, in order to have PyHESSIO reading the output of last sim_telarray version, I was forced to update the HESSIOxxx folder with the current one distributed by Konrad Bernloehr along with the latest CoRSiKa+sim_tel distribution, otherwise not being able to do it. Could the reported issue derive from some kind of mismatch between PyHESSIO and the latest HESSIOxxx?

[kosack]

Sounds like a definite bug somewhere. Is this true for prod2 and prod3 input files? Perhaps it was only tested with MC events...

In any case, we've switched to using PyEventIO instead of pyhessio in the latest version of ctapipe, which should do it correctly (and requires no compilation, no hessioxxx, or large memory footprint). So this package may be deprecated soon, unless there is more manpower to maintain it.

[dneise]

pyeventio is pip installable, so if @FrancescoGSaturni would like to try it out he could just do:

pip install pyeventio

And then have a look at the event header similar to the first example here:

https://github.com/cta-observatory/pyeventio#first-example

[FrancescoGSaturni]

Sounds like a definite bug somewhere. Is this true for prod2 and prod3 input files? Perhaps it was only tested with MC events...

In any case, we've switched to using PyEventIO instead of pyhessio in the latest version of ctapipe, which should do it correctly (and requires no compilation, no hessioxxx, or large memory footprint). So this package may be deprecated soon, unless there is more manpower to maintain it.

Hi Karl & Dominik, many thanks for the answers beforehand. We have found the issue working on sim_telarray files for the ASTRI Prototype. Is it possible as a first test to obtain a Prod2 or Prod3 input to check if the bug still holds?

pyeventio is pip installable, so if @FrancescoGSaturni would like to try it out he could just do:

pip install pyeventio

And then have a look at the event header similar to the first example here:

https://github.com/cta-observatory/pyeventio#first-example

Second question: since I am dealing at the moment with an already full-written Python script, can you confirm me wether or not the PyEventIO module supports the same tasks and functions of PyHESSIO (i.e. same syntax, same calling sequences and arguments, etc)? It would be extremely helpful for us in order to decide how to go on.

Thanks again!

[maxnoe]

and requires no compilation

Actually we use quite a bit of cython optimizations, so a c compilation is still required.

Second question: since I am dealing at the moment with an already full-written Python script

Are you using pyhessio directly or through ctapipe.io.eventsource?
The ctapipe event source is a drop in replacement for the old one.
But pyeventio has not the same API as pyhessio, as it was implemented independently.

To get started, you can look into: https://github.com/cta-observatory/pyeventio/blob/master/examples/try_simtelfile.py

[FrancescoGSaturni]

and requires no compilation

Actually we use quite a bit of cython optimizations, so a c compilation is still required.

Second question: since I am dealing at the moment with an already full-written Python script

Are you using pyhessio directly or through ctapipe.io.eventsource?
The ctapipe event source is a drop in replacement for the old one.
But pyeventio has not the same API as pyhessio, as it was implemented independently.

To get started, you can look into: https://github.com/cta-observatory/pyeventio/blob/master/examples/try_simtelfile.py

I am currently using pyhessio directly (i.e. as a standalone from its previous GitHub repository). I am now trying to implement pyeventio, and apparently it can solve my issue with minor fixes to my current script. Thanks!