ROOT-HEAD fails with "cling interactive line includer >>>: fatal error: module file '[snip]/Vc.pcm' not found: module file not found"

[andresailer]

Check duplicate issues.

• Checked for duplicates

Description

Since yesterday, March 13 we observe errors in our tests using our installed nightly view.
And also on LXPLUS, see here for an error from our build system: https://lcgapp-services.cern.ch/cdash/testDetails.php?test=24046676&build=314678

Reproducer

On lxplus for example

source /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/latest/x86_64-el9-gcc13-dbg/setup.sh
root

gives

<<< cling interactive line includer >>>: fatal error: module file '/build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/Vc.pcm' not found: module file not found
<<< cling interactive line includer >>>: note: imported by module 'MathCore' in '/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Wed/ROOT/HEAD/x86_64-el9-gcc13-dbg/lib/MathCore.pcm'
Failed to load module MathCore
Failed to load module Hist
Failed to load module ROOTBrowsable
Failed to load module Unfold
Failed to load module RHTTPSniff
Failed to load module HistPainter
Failed to load module PyMVA
Failed to load module FitPanel
Failed to load module Unuran
Failed to load module Quadp
Failed to load module GeomPainter
Failed to load module Genetic
Failed to load module Eve
Failed to load module TreeViewer
Failed to load module Physics
Failed to load module ROOTTMVASofieParser
Failed to load module EG
Failed to load module Tree
Failed to load module HistFactory
Failed to load module Spectrum
Failed to load module Matrix
Failed to load module Html
Failed to load module Hist
Failed to load module GuiHtml
Failed to load module Gpad
Failed to load module TMVAGui
Failed to load module Postscript
Failed to load module ROOTEve
Failed to load module RGL
Failed to load module Fumili
Failed to load module Geom
Failed to load module RooFitMore
Failed to load module X3d
Failed to load module SPlot
Failed to load module Hbook
Failed to load module RooFit
Failed to load module RCsg
Failed to load module RooStats
Failed to load module RooFitRDataFrameHelpers
Failed to load module GeomBuilder
Failed to load module FITSIO
Failed to load module Ged
Failed to load module Recorder
Failed to load module FFTW
Failed to load module ROOTWebDisplay
Failed to load module GuiBld
Failed to load module RooFitCore
Failed to load module Gui
Failed to load module ROOTHistDraw
Failed to load module GX11TTF
Failed to load module ROOTTMVASofie
Failed to load module TMVAUtils
Failed to load module ASImage
Failed to load module MathMore
Failed to load module RooFitXRooFit
Failed to load module RooFitHS3
Failed to load module Foam
Failed to load module SpectrumPainter
Failed to load module Minuit2
Failed to load module MLP
Failed to load module ROOTDataFrame
Failed to load module GenVector
Failed to load module ROOTBrowserv7
Failed to load module Minuit
Failed to load module Graf3d
Failed to load module TMVA
Failed to load module ROOTNTupleUtil
Failed to load module ASImageGui
Failed to load module Graf
Failed to load module GX11
Failed to load module Gdml
Failed to load module ROOTTreeViewer
Failed to load module MathCore
Failed to load module Gviz3d
Failed to load module WebGui6
Failed to load module ROOTHist
Failed to load module TreePlayer
Failed to load module ROOTGeomViewer
Failed to load module ROOTFitPanelv7
Failed to load module Smatrix
root.exe: /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT/HEAD/interpreter/llvm-project/clang/lib/Lex/Lexer.cpp:4370: bool clang::Lexer::LexTokenInternal(clang::Token&, bool): Assertion `Result.is(tok::eof) && "Preprocessor did not set tok:eof"' failed.

According to strace previous version ROOT would first try the /build/... folder to find Vc.pcm, and then look parallel to other ROOT libraries (not sure how that is decided). But now only the build path is tried and nothing else.

Vc.pcm is present

 ls -l /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/latest/x86_64-el9-gcc13-dbg/lib/Vc.pcm
lrwxrwxrwx. 1 cvmfs cvmfs 93 Mar 13 12:58 /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/latest/x86_64-el9-gcc13-dbg/lib/Vc.pcm -> /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Wed/ROOT/HEAD/x86_64-el9-gcc13-dbg/lib/Vc.pcm

ROOT version

master

Installation method

lcgcmake from source: options https://gitlab.cern.ch/sft/lcgcmake/-/blob/master/projects/CMakeLists.txt?ref_type=heads#L82

Operating system

alma9, alma8, ubuntu20, ubuntu22

Additional context

No response

[bellenot]

I don't know how to debug this, but since it's related to modules and it looks like the errors we use to have on MacOS, could it be related to this commit (@vgvassilev @dpiparo)? At least it's the only change I found so far that can be related to this issue

[dpiparo]

Yes, this is likely the cause. @vgvassilev do you understand why this is happening?

[vgvassilev]

Yes, I am debugging a similar issue.

[andresailer]

We see more or less the same thing in dev4 now:
Ironically the "opt" build gives more of an error message than the debug build
opt: https://lcgapp-services.cern.ch/cdash/testDetails.php?test=24134444&build=315763
dbg: https://lcgapp-services.cern.ch/cdash/testDetails.php?test=24138113&build=315805

[andresailer]

Following #14974, we see no change in the error message today:

lxplus966:~$ source /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/latest/x86_64-el9-gcc11-opt/setup.sh 
lxplus966:~$ python -c "import ROOT"
<<< cling interactive line includer >>>: fatal error: module file '/build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/Vc.pcm' not found: module file not found
<<< cling interactive line includer >>>: note: imported by module 'MathCore' in '/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-el9-gcc11-opt/lib/MathCore.pcm'
Failed to load module MathCore
[snip]

(Some of the builds that you might see in our cdash could suffer from picking up Vc.pcm in the build location but from builds for other platforms...
which results in an error message like

<<< cling interactive line includer >>>: fatal error: module file '/build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/Vc.pcm' is out of date and needs to be rebuilt: signature mismatch

)

[dpiparo]

Hi @andresailer , do you have means to see if this issue affects 6.30 and 6.28 branches as well?

[andresailer]

Yes, it also affects dev4 (cf. #14964 (comment)) == 6.30-patches

[andresailer]

I don't have anything right now that builds 6.28-patches

[dpiparo]

thanks andre. The 2 branches, 6.28 and 6.30, are similar enough to assume that also 6.28 suffers from this issue.

[vgvassilev]

I cannot reproduce the issue locally, do you mind building tonight with -DCMAKE_BUILD_TYPE=Debug -DLLVM_BUILD_TYPE=Debug.

[dpiparo]

I guess the comment above is for @andresailer

[andresailer]

We don't do nightly builds on Sunday mornings. I am trying to get a special
/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3debug/latest/x86_64-el9-gcc13-dbg
build done now which has the CMAKE and LLVM Debug build type.

[andresailer]

@vgvassilev The installation with LLVM Debug is there now

$ source /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3debug/latest/x86_64-el9-gcc13-dbg/setup.sh
$ grep LLVM /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3debug/Sun/ROOT/HEAD/x86_64-el9-gcc13-dbg/logs/ROOT-HEAD-configure.log
-- Building LLVM in 'Debug' mode.
-- LLVM host triple: x86_64-unknown-linux-gnu
-- LLVM default target triple: x86_64-unknown-linux-gnu

[vgvassilev]

@andresailer, could you test this pr: #15043

[andresailer]

I put the commit into the dev3debug of today, and it fixes the issue with the modules.

[dpiparo]

thanks @andresailer I propose to close only after we see a round of successful LCG nightlies.

[dpiparo]

Relevant PRs have been merged:

• master: [runtime_cxxmodules] Allow for changing the modulemap name to match the overlay #15043
• v6-30-00-patches: [Backport 6.30][runtime_cxxmodules] Allow for changing the modulemap name to match the overlay #15050
  We'll see tomorrow how dev3 and dev4 go.

[andresailer]

Thanks! Things are looking good for us in dev3 and dev4.
(dev3debug is not going to get updated and will be removed from CVMFS again)

[dpiparo]

Thanks to you @andresailer !

[github-actions]

Hi @dpiparo, @bellenot,

It appears this issue is closed, but wasn't yet added to a project. Please add upcoming versions that will include the fix, or 'not applicable' otherwise.

Sincerely,
🤖

[github-actions]

Hi @dpiparo, @bellenot,

It appears this issue is closed, but wasn't yet added to a project. Please add upcoming versions that will include the fix, or 'not applicable' otherwise.

Sincerely,
🤖