Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

workload/ycsb: package failed under stress #34004

Closed
cockroach-teamcity opened this issue Jan 15, 2019 · 11 comments
Closed

workload/ycsb: package failed under stress #34004

cockroach-teamcity opened this issue Jan 15, 2019 · 11 comments
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/0c7e491a76142a67750649facbce798442f74d25

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=(unknown) PKG=github.com/cockroachdb/cockroach/pkg/workload/ycsb TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1094320&tab=buildLog

github.com/cockroachdb/cockroach/vendor/github.com/kr/pretty
github.com/cockroachdb/cockroach/pkg/util/retry
github.com/cockroachdb/cockroach/pkg/util/stringencoding
github.com/cockroachdb/cockroach/pkg/sql/privilege
github.com/cockroachdb/cockroach/pkg/util/pretty
github.com/cockroachdb/cockroach/pkg/util/ring
github.com/cockroachdb/cockroach/pkg/util/timeofday
github.com/cockroachdb/cockroach/vendor/golang.org/x/text/internal/tag
github.com/cockroachdb/cockroach/vendor/golang.org/x/text/language
github.com/cockroachdb/cockroach/pkg/util/treeprinter
github.com/cockroachdb/cockroach/vendor/github.com/prometheus/client_golang/prometheus
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc
github.com/cockroachdb/cockroach/vendor/golang.org/x/text/internal/colltab
github.com/cockroachdb/cockroach/vendor/golang.org/x/text/collate
github.com/cockroachdb/cockroach/vendor/github.com/prometheus/client_golang/prometheus/graphite
github.com/cockroachdb/cockroach/vendor/github.com/lightstep/lightstep-tracer-go/collectorpb
github.com/cockroachdb/cockroach/vendor/github.com/lightstep/lightstep-tracer-go
github.com/cockroachdb/cockroach/vendor/github.com/Shopify/sarama
github.com/cockroachdb/cockroach/vendor/github.com/openzipkin-contrib/zipkin-go-opentracing
github.com/cockroachdb/cockroach/pkg/util/tracing
github.com/cockroachdb/cockroach/pkg/util/log
github.com/cockroachdb/cockroach/pkg/util/hlc
github.com/cockroachdb/cockroach/pkg/util/protoutil
github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgerror
github.com/cockroachdb/cockroach/pkg/util/interval
github.com/cockroachdb/cockroach/pkg/util/metric
github.com/cockroachdb/cockroach/pkg/util/ipaddr
github.com/cockroachdb/cockroach/pkg/server/telemetry
github.com/cockroachdb/cockroach/pkg/sql/sessiondata
github.com/cockroachdb/cockroach/pkg/sql/sem/types
github.com/cockroachdb/cockroach/pkg/util/timeutil/pgdate
github.com/cockroachdb/cockroach/pkg/storage/engine/enginepb
github.com/cockroachdb/cockroach/pkg/sql/lex
github.com/cockroachdb/cockroach/pkg/util/encoding
github.com/cockroachdb/cockroach/pkg/util/cache
github.com/cockroachdb/cockroach/pkg/sql/coltypes
github.com/cockroachdb/cockroach/pkg/util/json
github.com/cockroachdb/cockroach/pkg/roachpb
github.com/cockroachdb/cockroach/pkg/util/stop
github.com/cockroachdb/cockroach/pkg/settings/cluster
github.com/cockroachdb/cockroach/pkg/keys
github.com/cockroachdb/cockroach/pkg/security
github.com/cockroachdb/cockroach/pkg/util/mon
github.com/cockroachdb/cockroach/pkg/base
github.com/cockroachdb/cockroach/pkg/internal/client
github.com/cockroachdb/cockroach/pkg/sql/sem/tree
github.com/cockroachdb/cockroach/pkg/sql/sem/transform
github.com/cockroachdb/cockroach/pkg/sql/opt/cat
github.com/cockroachdb/cockroach/pkg/sql/parser
github.com/cockroachdb/cockroach/pkg/config
github.com/cockroachdb/cockroach/pkg/sql/sqlbase
github.com/cockroachdb/cockroach/pkg/sql/exec/types
github.com/cockroachdb/cockroach/pkg/sql/exec/execgen/cmd/execgen
execgen pkg/sql/exec/avg_agg.eg.go
execgen pkg/sql/exec/any_not_null_agg.eg.go
execgen pkg/sql/exec/hashjoiner.eg.go
execgen pkg/sql/exec/distinct.eg.go
execgen pkg/sql/exec/colvec.eg.go
execgen pkg/sql/exec/projection_ops.eg.go
execgen pkg/sql/exec/rowstovec.eg.go
execgen pkg/sql/exec/quicksort.eg.go
execgen pkg/sql/exec/selection_ops.eg.go
execgen pkg/sql/exec/sort.eg.go
execgen pkg/sql/exec/sum_agg.eg.go
go test -race -exec 'stress -maxruns 100 -maxfails 1 -stderr -p 4' -tags ' make x86_64_pc_linux_gnu' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v2.2.0-alpha.20181217-600-g0c7e491" -X "github.com/cockroachdb/cockroach/pkg/build.rev=0c7e491a76142a67750649facbce798442f74d25" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-pc-linux-gnu"  ' -run "." -timeout 0 github.com/cockroachdb/cockroach/pkg/workload/ycsb  -v -args -test.timeout 40m
go test: race flag may be set only once
run "go help test" or "go help testflag" for more information
Makefile:860: recipe for target 'stress' failed
make: *** [stress] Error 2

@cockroach-teamcity cockroach-teamcity added this to the 2.2 milestone Jan 15, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jan 15, 2019
@tbg tbg closed this as completed Jan 15, 2019
@tbg tbg reopened this Jan 15, 2019
@tbg
Copy link
Member

tbg commented Jan 15, 2019

Using this as an anchor for all the other issues like it.

I can repro this locally using

$ GOFLAGS='-race' go test -race ./pkg/storage/
go test: race flag may be set only once
run "go help test" or "go help testflag" for more information

I think go picks up the GOFLAGS variable, which has -race set, and then has another -race on the command line which it doesn't like (not sure why this just happened tonight, did we maybe bump the Go version in CI?)

@tbg
Copy link
Member

tbg commented Jan 15, 2019

That's probably it, GOFLAGS is new:

golang/go#27282

If this is true, we should "just" require Go 1.11 and stop passing GOFLAGS explicitly. Or we must prefix every invocation of go with GOFLAGS="".

@knz what do you think is the sane option here?

@knz
Copy link
Contributor

knz commented Jan 15, 2019

I thought peter had a PR to bump the req to 1.11? I'd vote for that then we can remove the explicit GOFLAGS.

@petermattis
Copy link
Collaborator

petermattis commented Jan 15, 2019 via email

@tbg
Copy link
Member

tbg commented Jan 15, 2019

Something about this is still janky. I just tried this out (i.e. not passing $(GOFLAGS) to go test ...) and it works if you specify make GOFLAGS=-race .... But what should also work is make stressrace PKG=... (which internally overrides GOFLAGS += -race) but this doesn't work -- go gets an empty GOFLAGS env var.

Perhaps someone with more Makefile knowledge can look at this. @petermattis?

@tbg
Copy link
Member

tbg commented Jan 15, 2019

(My preference right now would be making sure that the Makefile overrides GOFLAGS with an empty value initially, i.e. we don't let anything from the env slip in).

BTW I thought the non-working case that I describe above should work, according to https://ftp.gnu.org/old-gnu/Manuals/make-3.79.1/html_chapter/make_6.html#SEC68, though I don't even know whether I understood that correctly or whether it applies to our version of make.

@petermattis
Copy link
Collaborator

petermattis commented Jan 15, 2019 via email

@tbg
Copy link
Member

tbg commented Jan 15, 2019

Thanks @petermattis. I didn't save the branch, unfortunately but there really wasn't much to it.

@petermattis
Copy link
Collaborator

Note that cmd/teamcity-trigger/main.go is setting GOFLAGS=-race in order to get a stressrace build for the Cockroach_Nightlies_Stress. I suppose we can change that build config to look for a different environment variable than GOFLAGS and condition the build step to run either make stress or make stressrace.

@tbg
Copy link
Member

tbg commented Jan 15, 2019

I looked at that but it seemed nasty, also the general problem persists -- if you have GOFLAGS in your env, things will be weird and that's not good. It really seems like we want to make sure you don't pass GOFLAGS in through the environment. (Or rather, that make doesn't pass it on to go via the environment).

@petermattis
Copy link
Collaborator

GNU Make 3.82 provides a handy undefine function. But the make version on my computer is 3.81. Grrr.

We invoke go in some many places that this is a bit irritating to fix manually. Still looking.

petermattis added a commit to petermattis/cockroach that referenced this issue Jan 15, 2019
As of go1.11, `go` interprets the `GOFLAGS` env variable and errors out
if certain flags are duplicated in both the environment variable and on
the command line (e.g. `-race`).

Fixes cockroachdb#34004

Release note: None
craig bot pushed a commit that referenced this issue Jan 15, 2019
34013: build: avoid duplication of GOFLAGS for nightly stress r=jordanlewis,tbg a=petermattis

Clear the `GOFLAGS` env variable for the nightly stress invocations. As
of go1.11, `go` interprets that variable and errors out if certain flags
are duplicated in both the environment variable and on the command
line (e.g. `-race`).

Fixes #34004

Release note: None

Co-authored-by: Peter Mattis <petermattis@gmail.com>
@craig craig bot closed this as completed in #34013 Jan 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

No branches or pull requests

4 participants