roachtest: tpcc/nodes=3/w=max failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/137b282943695b8d2aabcbd5d69be85b58746acd

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=tpcc/nodes=3/w=max PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=933504&tab=buildLog

The test failed on master:
	test.go:498,cluster.go:890,tpcc.go:78,cluster.go:1199,errgroup.go:58: /home/agent/work/.go/bin/roachprod run teamcity-933504-tpcc-nodes-3-w-max:4 -- ./workload run tpcc --warehouses=1400 --duration=120m {pgurl:1-3} --histograms=logs/stats.json returned:
		stderr:
		
		stdout:
		            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		      1s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		      1s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		      1s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
		      1s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		      2s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		      2s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		      2s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		      2s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
		      2s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		Error: error in delivery: dial tcp 10.128.0.6:26257: socket: too many open files
		Error:  exit status 1
		: exit status 1
	test.go:498,cluster.go:1220,tpcc.go:81,tpcc.go:95: Goexit() was called

[tbg]

cc @nvanbenschoten :(

[nvanbenschoten]

Huh, I wonder what I'm missing?

cockroach git:(master) roachprod create -n=1 nathan-ulimit && roachprod run nathan-ulimit:1 -- 'ulimit -n && cat /proc/sys/fs/file-max'
Creating cluster nathan-ulimit with 1 nodes
OK
nathan-ulimit: [gce] 12h30m24s remaining
  nathan-ulimit-0001	nathan-ulimit-0001.us-east1-b.cockroach-ephemeral	10.142.0.2	35.237.75.68
Syncing...
nathan-ulimit: waiting for nodes to start 1/1
generating ssh key 1/1
distributing ssh key 0/0
retrieving hosts 1/1
scanning hosts 1/1
65536
1534448

I'll take a look reproducing tomorrow.

[tbg]

There's always something with ulimits. :)

[nvanbenschoten]

Let's give cockroachdb/roachprod#209 a try.