Streamlining DB12: Would you be impacted?

[aldbr]

The evolution of the DIRAC Benchmark (DB12) has become increasingly complex over time. Initially conceptualized to align with HEPSPEC06, subsequent developments have introduced various modifications and additional factors, complicating its application and maintenance.

TL;DR: We propose aligning DB12 exclusively with the LHCb HEPScore to simplify its maintenance and recalibration process. This change might affect how you use DB12, and we are interested in your feedback on this proposed alignment.

Key Developments:

1. Origin with HEPSPEC06 (DB12): DB12 was initially designed to mimic the HEPSPEC06 benchmark.
2. Introduction of DB16 for LHCb: Due to discrepancies observed in LHCb, DB16 was introduced, calculated as DB12 score / 0.65. This was only introduced in LHCbDIRAC.
3. Adjustments for Python3 Transition: With the transition to Python3, which affected DB12 scores, new factors were developed for both Intel and AMD to align Python3-based DB12 scores with those computed using Python2.
4. Alignment with LHCb HEPScore: Following WLCG's introduction of HEPScore as a replacement for HEPSPEC06, an additional factor is necessary to align DB12 scores with LHCb HEPScore.

Current Challenges:
The application of DB12 now faces complexities due to its detachment from a consistent benchmark basis and the constant need for adjustments:

• Each new version of the LHCb HEPScore necessitates a reanalysis of its impact on DB12.
• Updates in Python versions require a similar analysis concerning DB12.
• Currently, we still provide a way to disable to "correction factors" to retrieve the raw DB12 result.

Proposed Solution:
To simplify the process, we propose the following changes:

1. Python Version Freeze: Utilize Python 3.11 (at least until 2027), creating a dedicated Conda environment for the DIRAC benchmark. This environment would not be frequently updated, thus addressing concerns related to Python version changes.
2. Alignment Solely with LHCb HEPScore: By aligning the benchmark exclusively with HEPScore, we eliminate the need for multiple factors and recalibrations. Consequently, DB12 would only need reevaluation if there are changes in the LHCb HEPScore (or if the frozen version of Python becomes deprecated).

Each release would correspond to a LHCb HEPScore recalibration (the person in charge of the recalibration should use the latest Python version available at that time).
Concerning the DIRAC integration: the DIRAC benchmark would not run from the DIRAC mamba environment but from the DB12 mamba environment.

This approach would reduce the frequency of recalibrations, and provide a more stable and predictable benchmarking environment. However, it clearly aligns DB12 with a LHCb-specific benchmark (based on Monte-Carlo simulations).