key4hep release with k4SimDelphes tag v00-05

[bistapf]

For the purpose of preserving our FCC-hh analyses, having a stable release that matches our sample production would be important, meaning in detail:

• Including the latest changes to k4SimDelphes, as with the tag v00-05, specifically adding the support for storing the event weights and isolation variable in the conversion from Delphes output to edm4hep.
• If at all possible, ideally without the collection name updates in podio/edm4hep, so pre-(PR405)Make branch names for relations more legible for ROOT based I/O AIDASoft/podio#405 for the rest of the stack. If this is not an easy option, just an up-to-date release with all latest changes and the k4SimDelphes update would already be immensely helpful.

Posting this as an official request following up on the email discussion between @ataliercio , @hegner , @andresailer . As mentioned we have (very suboptimally) run the production using the nightly builds so far.

Let me know if any further information or action on our side is needed.

[andresailer]

Hi @bistapf,
I moved this request to the key4hep-stack repository, as making a release is more related to this repository.

What would be the urgency of this request?

[andresailer]

Regarding the k4SimDelphes Tag: You mean including this PR key4hep/k4SimDelphes#107 , right? That is part of the 00-05 tag.

[jmcarcell]

This is a non-trivial request so some clarifications will be needed. Since the purpose of this release is very specific and you have used the nightlies it would also be possible to use the exact versions you had in the nightlies, changing k4SimDelphes. Otherwise we can choose tagged versions and I think these would be the constraints:

• k4SimDelphes: v00-05 (any particular version of delphes that you want?)
• For podio the PR you mention isn't in any tag yet so we would get the latest version v00-16-06
• Same for edm4hep , we would get v00-10. Note here that we have to make sure that in the tags for other repos all have the TPCHit -> RawTimeSeries changes (Update schema diagram EDM4hep#206) and the subDetectorHitNumbers change (Change in edm4hep.yaml for consistent naming EDM4hep#188)
• For k4FWCore v01-00pre16, which is a bit old, from December 2022. The main change since then is that now podio Frames can be read using Gaudi, do you need this (probably not since it's very recent) ? If not we have to pin DD4hep to 1.23 and make sure LCIOConversions: fix attaching SimCaloHit collections to event AIDASoft/DD4hep#1019 is patched so that it doesn't write Frames.
  I think these would be the main points to take into account if choosing tagged versions but I may be missing something. Can you say if you need or don't need anything else (features, PRs, etc)? This is why if you used a nighty then at least that is tested so we could use that instead of some combination of tags and cherry picks that may or may not work

[bistapf]

Hi @andresailer @jmcarcell,

Thanks for your feedback.

What would be the urgency of this request?

For pratical purposes, not so super urgent since we are able to run over all the versions of files we produced (just with slightly different code). However, for documentation + analysis preservation purposes we think it would be important to resolve this soon, ideally before the end of the month (if that is possible at all, given that it's summer break time).

Regarding the k4SimDelphes Tag: You mean including this PR key4hep/k4SimDelphes#107 , right? That is part of the 00-05 tag.

Yes, this one and also the one that came before to introduce the event weights of course (PR106).

Since the purpose of this release is very specific and you have used the nightlies it would also be possible to use the exact versions you had in the nightlies, changing k4SimDelphes.

We last used the nightlies to produce samples in consistent format at the end of May/early June (I think 6th of June was the last date). Those nightlies are no longer around on cfmvs, right? But if it is easy to track which versions they contained, perhaps that is the best solution?

k4SimDelphes: v00-05 (any particular version of delphes that you want?)
Yes on k4SimDelphes, as clarified above also which PRs are important. For the Delphes version I think there is no particular requirement, the one that came with the nightlies in early June. Can I check this somehow in my files, do you know (the logs don't specify it)?

For k4FWCore v01-00pre16, which is a bit old, from December 2022. The main change since then is that now podio Frames can be read using Gaudi, do you need this (probably not since it's very recent) ?

No, we do not need this.

I think these would be the main points to take into account if choosing tagged versions but I may be missing something. Can you say if you need or don't need anything else (features, PRs, etc)? This is why if you used a nighty then at least that is tested so we could use that instead of some combination of tags and cherry picks that may or may not work

I don't think there are other features/PRs we needed, only the k4SimDelphes developments to support event weights and isolation variable in the conversion. There was an older change to have it support Pythia's resonance decay filter that we rely on, but that should come with the tag v00-05 as well.

For the nightlies, if there is an older version (~late May/early June) archived somewhere I can try if it fits our requirements easily. We tried with the oldest currently available ones on cfmvs (from late June) and they were "too new" (meaning the collection name change is alreadly implemented). Edited to add: Checking the logs from the production, those jobs ran with source /cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-05/x86_64-centos7-gcc12.2.0-opt/key4hep-stack/2023-06-05-ofh7qo/setup.sh still.

If it is possible at all to use something that doesn't introduce this yet, it would be the most ideal for this iteration of the FCC-hh analyses (and we will ensure to stay in touch better for any further iterations to avoid such issues, of course).

I hope this clarifies some things, please let me know if you need any other information. Thanks for your help!

[jmcarcell]

There is a release on cvmfs that has v00-05 of k4SimDelphes, has a commit of podio previous to that rename and should be very close to what you got with the 2023-06-05 nightly. Please give it a try:

source /cvmfs/sw.hsf.org/key4hep/releases/2023-06-05-fcchh/x86_64-centos7-gcc12.2.0-opt/key4hep-stack/2023-08-28-hsn6vj/setup.sh

[bistapf]

Dear @jmcarcell ,

sorry for the delay, but I have tested the release properly now and everything looks good and runs fine. Thank you very much!
So provided the release stays available to us (under the same path?) , I think we can consider this issue solved.
Thanks a lot again!

Birgit

[jmcarcell]

under the same path?

Yes, it will remain under the same path