roachprod: Put occasionally fails with scp error

[renatolabs]

We've been seeing an increased rate of scp related errors recently when running roachtests. It seems like this is especially more pronounced when roachtest is running multiple tests in parallel (and the test runner is copying large binaries to multiple clusters at the same time).

According to @yuzefovich, the issue can happen with very high probability when running the following command:

./bin/roachtest run -u yahor tpch_concurrency\$\$ --cockroach=artifacts/cockroach-short --workload=artifacts/workload --count=25 -p=5 --cpu-quota=1024

This has also been observed in roachtest failures, although at a lower rate (e.g., #91204).

Slack discussion (internal)

Updates:

• Issue seems to happen mostly when scp'ing from OSX to the roachprod cluster. Happens much less frequently when running a test from a gceworker.

Jira issue: CRDB-21419

[blathers-crl]

cc @cockroachdb/test-eng

[smg260]

I didn’t encounter any scp errors running the tpch_concurrency roachtest above from an m1 or gce-worker. Even tried just a basic roachtest which just did (over and over)

c.Put(ctx, t.Cockroach(), "./cockroach", c.Range(1, 4-1))
c.Put(ctx, t.DeprecatedWorkload(), "./workload", c.Node(4))

There have, however, been scp flakes in a few other roachtests recently.

@yuzefovich could there be some interfering configuration on your machine?

[yuzefovich]

Hm, I'm not sure, I don't think I have much custom configuration set up. I'll send you in a DM my bash config but otherwise I didn't make any changes.

[yuzefovich]

Today I’m in the SF office, and I’m no longer hitting the problems with uploading binaries from my mac - it seems like the problem is likely issues with my ISP (comcast) - I tried a test with 20 concurrent runs, and all uploads were successful.

[srosenberg]

@smg260 Are we still seeing these or has "ssh retries" eliminated these flakes?

[smg260]

This can be closed. Problem was isolated to the ISP/home network. In any case, SCP retries are implemented too.