Add jobs for CoreDNS scalability tests

[chrisohaver]

Adds Cluster DNS tests to scalability jobs (see proposed tests in kubernetes/kubernetes#63820)
Adds CoreDNS versions of the existing scalability jobs.

This is done to satisfy related graduation criteria of the KEP kubernetes/community#1956. In that KEP, CoreDNS becomes a GA alternative to kube-dns, and therefore it should be subject to the same level of scalability testing as kube-dns.

[chrisohaver]

FYI, @bowei, @johnbelamaric, @fturib, @rajansandeep

[rajansandeep]

/ok-to-test

[BenTheElder]

/cc @shyamjvs @wojtek-t

[BenTheElder]

/cc @MrHohn

[k8s-ci-robot]

@chrisohaver: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-test-infra-gubernator	73907d5	link	/test pull-test-infra-gubernator

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[shyamjvs]

@chrisohaver I'm not sure if duplicating all our scalability jobs is the right thing to do, for following reasons:

• if we double our jobs every time we want to test a new feature, this is going to explode the no. of jobs (which isn't scalable across verticals)
• also the jobs you added would run on 2000/5000-node clusters (which is a huge amount of quota) and they'd fail due to conflict with our regular 5k-node jobs (which are release-blocking)

If possible, I'd instead suggest replacing kube-dns with core-dns in our existing jobs temporarily for testing (that'll make this much easier and viable).
Does that SGTY?

[BenTheElder]

/hold
Quotas for scalability testing (which is costly[!]) come from @shyamjvs's team, deferring there.

[chrisohaver]

If possible, I'd instead suggest replacing kube-dns with core-dns in our existing jobs temporarily for testing (that'll make this much easier and viable). Does that SGTY?

OK, the decision during KEP approval was to support both coredns and kube-dns for at least one release... So if we are only allowed to test one, then we'll leave it at testing kube-dns, then switch it if/when coredns becomes default.

[fturib]

@shyamjvs : As CoreDNS is GA for this release, I guess it would be good to have one way to verify that it does not impact somehow the performance test, let say when we suspect that something goes wrong in the DNS server.

I understand that, due to the cost, we would not like to run those tests all time. But maybe we could have a prepared job that we can run "manually" when needed ? (using a /test command)
Would that be ok ? and how we would do that - having "pre-submit" that are not run by default ?

[shyamjvs]

Sure, that sounds reasonable. Or alternatively we can do some one-off testing (by switching it on in our job for 1 or 2 runs) - so that we can find any issues early on.

[shyamjvs]

@fturib Just missed your comment before my reply, sorry. Yes, that sounds reasonable (see suggestion I gave above).
Another easy option is to modify this job (which currently lies around as a manual job) to use for your purpose (instead of creating new one).

[fturib]

@shyamjvs : I am not very familiar with all usages of the infra-test jobs.
if we go with your proposition to use the "ci-kubernetes-e2e-gce-large-manual-up", what should we do, how does it work to run that job ?

I mean:

• do you expect we modify this job permanently to include the CoreDns deployment ? or is it only temporary when we need to make a test ? In that case, that means we need to have the temporary change merged before using that job against a kubernetes/kubternetes PR - right ?

About usage of this specific "manual-up" test:

• What permission or level of privilege is needed to run that job ?
• As I understand that "manual" job, it keeps the cluster running until you run the counterpart job ci-kubernetes-e2e-gce-large-manual-down . How do you get the credentials to access the cluster to evaluate its status ?
• Is there a way to run that job against a specific commit of kubernetes/kubernetes ? In order to be able to reproduce an issue in the field ?

[fturib]

@shyamjvs : could you clarify the how to "modify the job" ? (see my questions in the comment above). Thanks!

[fturib]

In order to go ahead on this PerformanceDNS, I would propose to:

• re-submit this PR by only extending the current jobs with "PerformanceDNS" -> that will make run this test for kube-dns (as it is the default)
• create another PR with one job that allow to run that "scalability" test in manual mode for CoreDNS when needed. I would think to duplicate the definition of "pull-kubernetes-e2e-gce-large-performance-coredns".

@chrisohaver , @shyamjvs : are you ok with that proposition ?

[chrisohaver]

Of course thats OK with me. But I think more important that it's OK with @shyamjvs et al.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chrisohaver

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[chrisohaver]

FYI, I did not close this issue. I backed out the rejected changes, and pushed the branch.

I guess Github automatically closes a PR if it ever see's a commit with no deltas from master.

[shyamjvs]

@chrisohaver @fturib - Would you be able to attend today's sig-scalability meeting at 9:30 AM pacific time? It might be easier to reach a consensus there than on the github thread.

[fturib]

@shyamjvs : sure will attend.

[fturib]

@shyamjvs : we missed the link you pasted into the chat zoom. Could you paste them here ? Thanks!

[shyamjvs]

https://github.com/kubernetes/kubernetes/tree/master/test/integration/scheduler_perf - This is where we have our scheduler integration tests for benchmarking performance.