Broken V8 build on S390x

[targos]

https://ci.nodejs.org/job/node-test-commit-v8-linux/2674/nodes=rhel72-s390x,v8test=v8test/console

/cc @nodejs/platform-s390

10:50:33 FAILED: obj/third_party/zlib/zlib/trees.o 
10:50:33 gcc -MMD -MF obj/third_party/zlib/zlib/trees.o.d -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_NSS_CERTS=1 -DUSE_X11=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DZLIB_IMPLEMENTATION -I../.. -Igen -I../../third_party/zlib -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -funwind-tables -fPIC -pipe -pthread -m64 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -O2 -fno-ident -fdata-sections -ffunction-sections -fno-omit-frame-pointer -g0 -fvisibility=hidden -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -Wno-comments -Wno-packed-not-aligned -Wno-missing-field-initializers -Wno-unused-parameter -std=gnu11 -c ../../third_party/zlib/trees.c -o obj/third_party/zlib/zlib/trees.o
10:50:33 cc1plus: warning: command line option ‘-std=gnu11’ is valid for C/ObjC but not for C++
10:50:33 ../../third_party/zlib/trees.c:379:30: error: variable or field ‘Cr_z__tr_init’ declared void
10:50:33  void ZLIB_INTERNAL _tr_init(s)
10:50:33                               ^
10:50:33 ../../third_party/zlib/trees.c:379:29: error: ‘s’ was not declared in this scope
10:50:33  void ZLIB_INTERNAL _tr_init(s)
10:50:33                              ^
10:50:33 ../../third_party/zlib/trees.c:381:1: error: expected unqualified-id before ‘{’ token
10:50:33  {
10:50:33  ^
10:50:33 ../../third_party/zlib/trees.c:407:24: error: variable or field ‘init_block’ declared void
10:50:33  local void init_block(s)
10:50:33                         ^
10:50:33 ../../third_party/zlib/trees.c:407:23: error: ‘s’ was not declared in this scope
10:50:33  local void init_block(s)
10:50:33                        ^
10:50:33 ../../third_party/zlib/trees.c:409:1: error: expected unqualified-id before ‘{’ token
10:50:33  {
10:50:33  ^
10:50:33 ../../third_party/zlib/trees.c:451:23: error: variable or field ‘pqdownheap’ declared void
10:50:33  local void pqdownheap(s, tree, k)
10:50:33                        ^
10:50:33 ../../third_party/zlib/trees.c:451:23: error: ‘s’ was not declared in this scope
10:50:33 ../../third_party/zlib/trees.c:451:26: error: ‘tree’ was not declared in this scope
10:50:33  local void pqdownheap(s, tree, k)
10:50:33                           ^~~~
10:50:33 ../../third_party/zlib/trees.c:451:32: error: ‘k’ was not declared in this scope
10:50:33  local void pqdownheap(s, tree, k)
10:50:33                                 ^
10:50:33 ../../third_party/zlib/trees.c:455:1: error: expected unqualified-id before ‘{’ token
10:50:33  {
10:50:33  ^
10:50:33 cc1plus: warning: unrecognized command line option ‘-Wno-packed-not-aligned’

[mhdawson]

@miladfarca can you take a look.

[miladfarca]

Is it possible to clear any cached files? export 'CXX=ccache s390x-redhat-linux-g++' seems to be interfering with the dependencies. I unset CC and CXX and compiled it successfully.

[targos]

@nodejs/build ^

[richardlau]

Is it possible to clear any cached files? export 'CXX=ccache s390x-redhat-linux-g++' seems to be interfering with the dependencies. I unset CC and CXX and compiled it successfully.

On the machine you can run ccache -c or ccache --cleanup to wipe the cache completely. Maybe we could add some logic to select-compiler.sh to optionally (via an option on the Jenkins jobs) set the CCACHE_RECACHE environment variable which would ignore the stored cache but write the new results?

https://ccache.dev/manual/latest.html

[miladfarca]

Thanks @richardlau , Just ran the above command to clear cache. @targos is it possible to run the Jenkins job again.

[richardlau]

Started a rebuild: https://ci.nodejs.org/job/node-test-commit-v8-linux/2679/

[richardlau]

Still failed 😞 https://ci.nodejs.org/job/node-test-commit-v8-linux/2679/nodes=rhel72-s390x,v8test=v8test/console

[miladfarca]

I ran the command as root, does it have to be ran as iojs? is it possible to delete the files manually or even disable the cache on Jenkins as a test?

[richardlau]

Yes, it probably should be run as iojs (the default location of the cache is $HOME/.ccache). Also I misread the manual and we should run ccache -C (capital C)/ccache --clear to wipe the cache (--cleanup is housekeeping).

I've gone ahead and done so and kicked off another build:
https://ci.nodejs.org/job/node-test-commit-v8-linux/2681/nodes=rhel72-s390x,v8test=v8test/console

[richardlau]

Yes, it probably should be run as iojs (the default location of the cache is $HOME/.ccache). Also I misread the manual and we should run ccache -C (capital C)/ccache --clear to wipe the cache (--cleanup is housekeeping).

I've gone ahead and done so and kicked off another build:
https://ci.nodejs.org/job/node-test-commit-v8-linux/2681/nodes=rhel72-s390x,v8test=v8test/console

Still failed. I'm not even sure ccache is being used as I cleared it (with ccache -C) and ccache -s shows the cache size as 0.0 kB after the failed build.

[miladfarca]

As another try, could you also clear the Jenkins workspace. Want to make sure it is fully downloading the latest dependencies.

[richardlau]

As another try, could you also clear the Jenkins workspace. Want to make sure it is fully downloading the latest dependencies.

I already did for https://ci.nodejs.org/job/node-test-commit-v8-linux/2681/nodes=rhel72-s390x,v8test=v8test/console as https://ci.nodejs.org/job/node-test-commit-v8-linux/nodes=rhel72-s390x,v8test=v8test/2680/console errored with You have unstaged changes.

[miladfarca]

Seems like g++ is being used to compile part of the V8 dependencies where gcc should have been used. Looking a closer look at tools/make-v8.sh under s390x we have:

ln -s $CXX_PATH "$BUILD_TOOLS/gcc"

Which is causing the issue. Is gcc pointing to CXX (g++) for a specific reason? If not then I have made this PR to fix the issue: nodejs/node#30817

Turns out export 'CXX=ccache s390x-redhat-linux-g++' is setting the CXX flag which is causing the later events to happen.

[richardlau]

Which is causing the issue. Is gcc pointing to CXX (g++) for a specific reason? If not then I have made this PR to fix the issue: nodejs/node#30817

@miladfarca No idea -- I'm not even sure what purpose creating the symlinks in $BUILD_TOOLS serves but you probably understand the V8 build system better than I do.

In any case I've landed nodejs/node#30817.

cc @targos

[targos]

Last canary build is fine: https://ci.nodejs.org/job/node-test-commit-v8-linux/2687/

Thanks for the fix!

[mhdawson]

@miladfarca Thanks !