Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: acceptance/version-upgrade failed: workload crashes with Illegal instruction #58754

Closed
cockroach-teamcity opened this issue Jan 12, 2021 · 11 comments
Closed

roachtest: acceptance/version-upgrade failed: workload crashes with Illegal instruction #58754

cockroach-teamcity opened this issue Jan 12, 2021 · 11 comments
Assignees
@itsbilal @jbowens @RaduBerinde
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Milestone
20.1

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).acceptance/version-upgrade failed on release-20.1@e395c0c7c48a279334f0e94dfb7030a3eafa093f:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	cluster.go:2198,mixed_version_schemachange.go:46,versionupgrade.go:189,versionupgrade.go:177,acceptance.go:53,acceptance.go:90,test_runner.go:749: output in run_071812.863_n1-4_workload_init_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2574842-1610435721-15-n4cpu4:1-4 -- ./workload init schemachange returned: exit status 20
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2276
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2196
		  | main.uploadAndInitSchemaChangeWorkload.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:46
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runVersionUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:177
		  | main.registerAcceptance.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:53
		  | main.registerAcceptance.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:90
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_071812.863_n1-4_workload_init_schemachange
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2574842-1610435721-15-n4cpu4:1-4 -- ./workload init schemachange returned
		  | stderr:
		  | Error: COMMAND_PROBLEM: exit status 132
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload init schemachange
		  |   | ```
		  | Wraps: (3) exit status 132
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | teamcity-2574842-1610435721-15-n4cpu4: ./workload init schemachange
		  |    1: bash: line 1:  4493 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		  |    2: bash: line 1:  4245 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		  |    3: bash: line 1:  4156 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		  |    4: bash: line 1:  4206 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError
More

Artifacts: /acceptance/version-upgrade
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-release-20.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jan 12, 2021
@cockroach-teamcity cockroach-teamcity added this to the 20.1 milestone Jan 12, 2021
@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 12, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).acceptance/version-upgrade failed on release-20.1@5197d58536a02e2f9170aee340dc1a70c87ead7a:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	cluster.go:2198,mixed_version_schemachange.go:46,versionupgrade.go:189,versionupgrade.go:177,acceptance.go:53,acceptance.go:90,test_runner.go:749: output in run_071854.530_n1-4_workload_init_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2578476-1610522081-11-n4cpu4:1-4 -- ./workload init schemachange returned: exit status 20
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2276
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2196
		  | main.uploadAndInitSchemaChangeWorkload.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:46
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runVersionUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:177
		  | main.registerAcceptance.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:53
		  | main.registerAcceptance.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:90
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_071854.530_n1-4_workload_init_schemachange
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2578476-1610522081-11-n4cpu4:1-4 -- ./workload init schemachange returned
		  | stderr:
		  | Error: COMMAND_PROBLEM: exit status 132
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload init schemachange
		  |   | ```
		  | Wraps: (3) exit status 132
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | teamcity-2578476-1610522081-11-n4cpu4: ./workload init schemachange
		  |    1: bash: line 1:  4612 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		  |    2: bash: line 1:  4280 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		  |    3: bash: line 1:  4167 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		  |    4: bash: line 1:  4231 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError
More

Artifacts: /acceptance/version-upgrade
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@RaduBerinde RaduBerinde changed the title roachtest: acceptance/version-upgrade failed roachtest: acceptance/version-upgrade failed: workload crashes with Illegal instruction Jan 13, 2021
@RaduBerinde
Copy link
Member

I've been trying to reproduce, no luck so far.

@cockroach-teamcity
Copy link
Member Author

(roachtest).acceptance/version-upgrade failed on release-20.1@217be74f52b63537110775d79a423a27cfa06c4d:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	cluster.go:2198,mixed_version_schemachange.go:46,versionupgrade.go:189,versionupgrade.go:177,acceptance.go:53,acceptance.go:90,test_runner.go:749: output in run_030358.183_n1-4_workload_init_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2582001-1610593279-09-n4cpu4:1-4 -- ./workload init schemachange returned: exit status 20
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2276
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2196
		  | main.uploadAndInitSchemaChangeWorkload.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:46
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runVersionUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:177
		  | main.registerAcceptance.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:53
		  | main.registerAcceptance.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:90
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_030358.183_n1-4_workload_init_schemachange
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2582001-1610593279-09-n4cpu4:1-4 -- ./workload init schemachange returned
		  | stderr:
		  | Error: COMMAND_PROBLEM: exit status 132
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload init schemachange
		  |   | ```
		  | Wraps: (3) exit status 132
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | teamcity-2582001-1610593279-09-n4cpu4: ./workload init schemachange
		  |    1: bash: line 1:  4747 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		  |    2: bash: line 1:  4251 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		  |    3: bash: line 1:  4152 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		  |    4: bash: line 1:  4198 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError
More

Artifacts: /acceptance/version-upgrade
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@RaduBerinde RaduBerinde self-assigned this Jan 14, 2021
@RaduBerinde
Copy link
Member

Started a run with debug=on: https://teamcity.cockroachdb.com/viewLog.html?buildId=2582001&

This is what I get on one left over hosts with gdb ./workload:

Program received signal SIGILL, Illegal instruction.
rocksdb::BlockBasedTableFactory::BlockBasedTableFactory ()
    at /home/agent/work/.go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/block_based_table_factory.cc:175
175	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/block_based_table_factory.cc: No such file or directory.

@RaduBerinde
Copy link
Member

Line of code:

175     table_options_.block_cache = NewLRUCache(8 << 20);

Offending instruction:

   0x00000000037ae270 <+672>:	cmpq   $0x0,0x30(%rbx)
   0x00000000037ae275 <+677>:	jne    0x37ae20e <rocksdb::BlockBasedTableFactory::BlockBasedTableFactory(rocksdb::BlockBasedTableOptions const&)+574>
=> 0x00000000037ae277 <+679>:	vpxord %zmm0,%zmm0,%zmm0

@RaduBerinde
Copy link
Member

vpxord is part of the AVX512 instruction set. Very surprising that this is used by default.. Maybe we want to pass -mno-avx512f to gcc for rocksdb. I am assigning this to storage for now since they would be most knowledgeable in rocksdb.

@RaduBerinde
Copy link
Member

Or perhaps gcc is misconfigured in the builder container. Although the flags shown by gcc -v in this image (cockroachdb/builder:20201209-060719) look fine:

roach@48dddcdc55e3:/go$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.12' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12)

@RaduBerinde
Copy link
Member

I think the problem is this line in rocksdb/build_tools/build_detect_platform:

543     COMMON_FLAGS="$COMMON_FLAGS -march=native "

-march=native means that it will generate code that may only work on the processor of this machine. There is no reason to have native anywhere.

I noticed some march=native in cryptopp as well..

@cockroach-teamcity
Copy link
Member Author

(roachtest).acceptance/version-upgrade failed on release-20.1@217be74f52b63537110775d79a423a27cfa06c4d:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	cluster.go:2198,mixed_version_schemachange.go:46,versionupgrade.go:189,versionupgrade.go:177,acceptance.go:53,acceptance.go:90,test_runner.go:749: output in run_071913.751_n1-4_workload_init_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2583445-1610608490-10-n4cpu4:1-4 -- ./workload init schemachange returned: exit status 20
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2276
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2196
		  | main.uploadAndInitSchemaChangeWorkload.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:46
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runVersionUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:177
		  | main.registerAcceptance.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:53
		  | main.registerAcceptance.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:90
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_071913.751_n1-4_workload_init_schemachange
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2583445-1610608490-10-n4cpu4:1-4 -- ./workload init schemachange returned
		  | stderr:
		  | Error: COMMAND_PROBLEM: exit status 132
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload init schemachange
		  |   | ```
		  | Wraps: (3) exit status 132
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | teamcity-2583445-1610608490-10-n4cpu4: ./workload init schemachange
		  |    1: bash: line 1:  4598 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		  |    2: bash: line 1:  4419 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		  |    3: bash: line 1:  4263 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		  |    4: bash: line 1:  4276 Illegal instruction     (core dumped) bash -c "./workload init schemachange"
		  | COMMAND_PROBLEM: exit status 132
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError
More

Artifacts: /acceptance/version-upgrade
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@petermattis
Copy link
Collaborator

When building release binaries, we pass PORTABLE=1 to RocksDB to disable -march=native. workload is built as a non-release binary which is what is causing -march=native to be passed. Let me whip up a PR to always pass PORTABLE=1.

@petermattis petermattis mentioned this issue Jan 14, 2021
Merged
petermattis added a commit to petermattis/cockroach that referenced this issue Jan 14, 2021
@petermattis
build: always specify PORTABLE=1 when building RocksDB
893c7fc 
Something recently changed on the TC agents (presumably usage of a new
machine type with a different CPU) so that using `-march=native` when
building RocksDB causes AVX512 instructions to be used which are not
present on roachprod machines. The result is that binaries built in this
way will die with "illegal instruction". RocksDB has a mechanism to
disable the use of `-march=native`: pass `PORTABLE=1` to the make
command. We were already doing this for release builds. Now we're doing
it for all builds.

Fixes cockroachdb#58754

Release note: none
petermattis added a commit to petermattis/cockroach that referenced this issue Jan 14, 2021
@petermattis
build: always specify PORTABLE=1 when building RocksDB
8b75a7e 
Something recently changed on the TC agents (presumably usage of a new
machine type with a different CPU) so that using `-march=native` when
building RocksDB causes AVX512 instructions to be used which are not
present on roachprod machines. The result is that binaries built in this
way will die with "illegal instruction". RocksDB has a mechanism to
disable the use of `-march=native`: pass `PORTABLE=1` to the make
command. We were already doing this for release builds. Now we're doing
it for all builds.

Fixes cockroachdb#58754

Release note: none
@RaduBerinde
Copy link
Member

#58997 fixed this, this test passed in https://teamcity.cockroachdb.com/viewLog.html?buildId=2584467&

@petermattis petermattis mentioned this issue Jan 14, 2021
Merged
@petermattis petermattis mentioned this issue Jan 14, 2021
Merged
petermattis added a commit to petermattis/cockroach that referenced this issue Jan 15, 2021
@petermattis
build: always specify PORTABLE=1 when building RocksDB
61a8890 
Something recently changed on the TC agents (presumably usage of a new
machine type with a different CPU) so that using `-march=native` when
building RocksDB causes AVX512 instructions to be used which are not
present on roachprod machines. The result is that binaries built in this
way will die with "illegal instruction". RocksDB has a mechanism to
disable the use of `-march=native`: pass `PORTABLE=1` to the make
command. We were already doing this for release builds. Now we're doing
it for all builds.

Fixes cockroachdb#58754

Release note: none
petermattis added a commit that referenced this issue Jan 15, 2021
@petermattis
build: always specify PORTABLE=1 when building RocksDB
973602e 
Something recently changed on the TC agents (presumably usage of a new
machine type with a different CPU) so that using `-march=native` when
building RocksDB causes AVX512 instructions to be used which are not
present on roachprod machines. The result is that binaries built in this
way will die with "illegal instruction". RocksDB has a mechanism to
disable the use of `-march=native`: pass `PORTABLE=1` to the make
command. We were already doing this for release builds. Now we're doing
it for all builds.

Fixes #58754

Release note: none
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@itsbilal itsbilal

@jbowens jbowens

@RaduBerinde RaduBerinde

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Projects
None yet
Milestone
20.1
Development

No branches or pull requests
5 participants
@itsbilal @jbowens @petermattis @RaduBerinde @cockroach-teamcity