Setup greenhouse in k8s-infra-prow-build cluster

[spiffxp]

Greenhouse is a bazel remote caching setup used to provide faster build times for jobs that use bazel. PR jobs get the most benefit.

In the google.com build cluster (k8s-prow-builds/prow) it's deployed to a nodepool reserved specifically for it, tuned for max IOPS

• node: n1-standard-32 (ref: https://cloud.google.com/compute/docs/disks/performance#machine-type-disk-limits)
• volume: 3TB pd-ssd (ref: https://cloud.google.com/compute/docs/disks/performance#ssd_persistent_disk_performance_by_disk_size)

ref: #752, followup to: #830

[spiffxp]

The n1-standard-32 puts k8s-infra-prow-builds over its CPU quota

Looking at us-central1, here are the differences between k8s-infra-prow-build vs. k8s-prow-builds:

quota	k8s-infra in-use	k8s-infra quota	goog in-use	goog quota
CPUs	48	72	1296	1500
In-use IP Addresses	6	69	159	250
Persistent Disk SSD (TB)	1.5	20.5	42.6	46

[spiffxp]

Submitted a quota request for 256 CPUs: current cluster mean CPU is ~40%, which works out to ~518, I chose a nearby power of two and cut it in half. Works out to 28 n1-highmem-8 nodes + the n1-standard-32 for greenhouse.

[spiffxp]

Right. k8s-infra-prow-build is a regional cluster. So if I make a nodepool of size 1, it's actually 3 nodes, one in each zone. I don't think we want three greenhouses.

I could:

• A) Use gcloud to restrict the node locations to a single zone... so the master is regional but the nodepool isn't (not 100% sure this is possible)
• B) Live with the 3 nodes, use 2 as regular build cluster nodes, and apply the dedicated:greenhouse label and associated taint to the other one manually using kubectl. This does mean the two zones where the greenhouse instance doesn't live are going to incur egress-between-zones cost
• C) Redo k8s-infra-prow-build as a zonal cluster

I'm leaning B

[spiffxp]

/assign @spiffxp
I'm working on it

/assign @thockin @BenTheElder
WDYT?

[ameukam]

Can I suggest :

• D) Add a variable to k8s-infra-gke-nodepool that represent node_count, set it to 1 and set the location of the module to a zone (us-central1-f)

I can't really test this but can work on it once #830 is merged.

[spiffxp]

Unfortunately I tried D (force the location to a zone), terraform plan is happy, but terraform apply yields:
Error: error creating NodePool: googleapi: Error 404: Not found: projects/k8s-infra-prow-build/zones/us-central1-b/clusters/prow-build., notFound

[spiffxp]

Have greenhouse pod request 32 cores

[thockin]

31-point-something to be fair :)

…

On Wed, May 13, 2020 at 8:59 AM Aaron Crickenberger ***@***.***> wrote: Have greenhouse pod request 32 cores — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.

[spiffxp]

I have it up and running and am waiting for #830 to land

[spiffxp]

I've opened #885

Need to confirm greenhouse-metrics service is setup properly

[spiffxp]

/close

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.