dl1/event/telescope/trigger written for events that should have been removed by the SoftwareTrigger

[maxnoe]

Describe the bug

Reported by @arrabito for the processing of the Prod6 divergent pointing data on the GRID,
which due to a bug in the simulation config does not contain the hardware stereo trigger.

grid lfns:

/vo.cta.in2p3.fr/MC/PROD6/Paranal/gamma-diffuse/ctapipe-merge/4111/Data/-10xxx/gamma_20deg_0deg___cta-prod6-paranal-2147m-Paranal-divergent-div0.0043-dark_cone10.beta4lst_train_en_merged.DL2.h5

The telescope trigger table contains more events than the parameters table:

from ctapipe.io import read_table
path = "gamma_20deg_0deg___cta-prod5b-lapalma_desert-2158m-LaPalma-dark_cone10.alpha_train_cl_applied.DL2.h5"

trigger = read_table(path, "/dl1/event/telescope/trigger", condition="tel_id==1")
parameters = read_table(path, "/dl1/event/telescope/parameters/tel_001")

print(len(trigger))
print(len(parameters))

145709
145565

All additional rows contain the default / nan values in the trigger data:

obs_id event_id tel_id time n_trigger_pixels
------ -------- ------ ---- ----------------
    15     2813      1  0.0               -1
    28   187615      1  0.0               -1
    47    32003      1  0.0               -1
    54    36405      1  0.0               -1
   113    96111      1  0.0               -1
   119    40813      1  0.0               -1
   140    93814      1  0.0               -1
   146   225116      1  0.0               -1
   152   140105      1  0.0               -1
   154    15407      1  0.0               -1
   161   510707      1  0.0               -1
   168    97907      1  0.0               -1
   172   140117      1  0.0               -1
   178    54914      1  0.0               -1
   186    10413      1  0.0               -1
   194    72809      1  0.0               -1
   239   201409      1  0.0               -1
   327   359406      1  0.0               -1
   363    27602      1  0.0               -1
   368    59907      1  0.0               -1
   ...      ...    ...  ...              ...
  1716   120812      1  0.0               -1
  1729   383419      1  0.0               -1
  1742    59812      1  0.0               -1
  1747    13010      1  0.0               -1
  1761    43808      1  0.0               -1
  1763    38610      1  0.0               -1
  1765    59410      1  0.0               -1
  1769   109210      1  0.0               -1
  1782    88509      1  0.0               -1
  1784    68917      1  0.0               -1
  1789   184011      1  0.0               -1
  1790   101702      1  0.0               -1
  1823     4814      1  0.0               -1
  1831      310      1  0.0               -1
  1832    55903      1  0.0               -1
  1851   188704      1  0.0               -1
  1863    77111      1  0.0               -1
  1870    89010      1  0.0               -1
  1905     2214      1  0.0               -1
  1932    21305      1  0.0               -1

So this might be an issue somewhere else, since the SoftwareTrigger seems to clear the trigger data correctly, but then the DataWriter? for some reason writes the default values.

To Reproduce

Tried to reproduce using the prod3 divergent file in the test data, but no success.

Expected behavior

Tables have same length and removed telescopes do not show up in the trigger table.

[maxnoe]

This is an unfortunate combination of several things, connected to several open issues about general design changes:

1. the SoftwareTrigger did not clear everything connected to the telescope event (CEP2!),
   e.g. not simulation and important for this bug not pointing (See DataWriter should not write pointing info from each event #1562)
2. The DataWriter writes the pointing using the timestamp from the trigger, that was cleared, so accessing event.trigger.tel[tel_id].time filled the default container due to Map being a defaultdict (See Remove defaultdict behavior from map, check type of value #2084).
3. The DataWriter then wrote the newly filled default container

[mdpunch]

@kosack I think this is related to our F2F discussion.

I noted that if the events with no coincidence from the software trigger (SWAT) are removed in real data, then:

• It will be difficult to find the single-telescope trigger rates
• It will be difficult to calculate the array dead-time (if we ever figure out how this should be done)

So, in the DL0 data there should be some trace of the non-coincident camera triggers, which would require a change in the Data Model. I also think the MC's should follow the same change (but that may be just me).

[maxnoe]

@mdpunch this is a quick fix for something we implemented for the analysis of simulated data a bit incorrectly.

I would suggest to open an issue if larger changes are needed, which it sounds like

[mdpunch]

Okay, I will leave that to Karl @kosack

[maxnoe]

If it affects the CTAO data model, we here would wait for a new release of that after the change request has gone through.