resultstore-upload job is continually failing

[spiffxp]

What happened:
https://testgrid.k8s.io/sig-testing-prow#resultstore-upload is continually failing

What you expected to happen:
for it not to fail

Please provide links to example occurrences, if any:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-test-infra-resultstore-upload/1164243772695908355

Anything else we need to know?:
It's not clear to me whether we rely on this and so I suggest at a minimum we stop running this 280 times a day

/assign @fejta

[spiffxp]

Opened #14010 to slow the job down

[fejta]

From the logs:

time="2019-08-21T18:34:55Z" level=fatal msg="Failed transfer" error="transfer: 3 errors transferring 957 groups: transfer ci-kubernetes-e2e-gci-gce-coredns-nodecache in gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-coredns-nodecache: gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-coredns-nodecache/1164160204770643969/: download: suites: parse: not valid testsuites: expected element type <testsuites> but have <testsuite> nor testsuite: XML syntax error on line 346: unescaped < inside quoted string, transfer ci-kubernetes-e2e-gce-scale-correctness in gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness: gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/1164145604574580738/: upload https://source.cloud.google.com/results/invocations/3930d827-cbd5-4d0b-ac37-569c31c10ab6/targets/test: create action: rpc error: code = InvalidArgument desc = Request contains an invalid argument., transfer ci-kubernetes-e2e-gce-scale-performance in gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance: gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/1163858705188392960/: upload https://source.cloud.google.com/results/invocations/4cecdc0b-8121-4cc7-ba6d-309370d20965/targets/test: create action: rpc error: code = InvalidArgument desc = Request contains an invalid argument."

A better thing to do here would be to delete the ci-kubernetes-e2e-gce-scale-correctness job and its invalid xml output.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[spiffxp]

/remove-lifecycle rotten
it's still failing on the same jobs, I can't tell at a glance what it is about their junit*.xml files that's invalid

[fejta]

#16645 is one thing

another thing looks to be

{"component":"storeship","error":"transfer: 2 errors transferring 1213 groups: transfer ci-kubernetes-e2e-gce-scale-correctness in gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness: gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/1235535977896742912/: upload https://source.cloud.google.com/results/invocations/49068598-d9b1-4fab-a6e6-6168e35a40fd/targets/test: create action: rpc error: code = InvalidArgument desc = Request contains an invalid argument., transfer ci-kubernetes-e2e-gce-scale-performance in gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance: gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/1235611477314899968/: upload https://source.cloud.google.com/results/invocations/5b220db3-cc10-4d41-8434-fd52dc240c31/targets/test: create action: rpc error: code = InvalidArgument desc = Request contains an invalid argument.","file":"experiment/resultstore/main.go:114","func":"main.main","level":"fatal","msg":"Failed transfer","time":"2020-03-06T06:24:49Z"}

ci-kubernetes-e2e-gce-scale-correctness
ci-kubernetes-e2e-gce-scale-performance

[fejta]

I believe this is because these jobs have hundreds of thousands of artifact files.

[fejta]

This miserly service appears to only let me attach ~40,000-50,000 files per test result

[spiffxp]

So we're now down to whatever is going on with e2e-kops-aws-cni-amazon-vpc/

{
  "component": "resultstore",
  "error": "gs://kubernetes-jenkins/logs/e2e-kops-aws-cni-amazon-vpc/1248138660340043779/: download: suites: parse: not valid testsuites: expected element type <testsuites> but have <testsuite> nor testsuite: XML syntax error on line 792: illegal character code U+0000",
  "file": "experiment/resultstore/main.go:307",
  "func": "main.transfer.func1.1",
  "level": "error",
  "msg": "Transfer failed",
  "time": "2020-04-10T03:07:12Z"
}

[fejta]

https://testgrid.k8s.io/sig-testing-prow#resultstore-upload&width=20