-
Notifications
You must be signed in to change notification settings - Fork 791
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add infra scripts/terraform for prow build cluster prototype #806
Conversation
2984b9c
to
7ff84fb
Compare
/uncc @mikedanese |
I have already actuated the stuff in this PR, so I'm running audit now to open a separate PR catching these changes (and anything else that may have happened) |
#807 is the audit |
specifically: - copied aaa and renamed - followed IMPORTANT comments for test cluster EXCEPT - used n1-highmem-2 because e2e jobs eat memory like candy and were getting evicted from n1-standard-1's
address terraform plan warnings, so: - no needless interpolation - got rid of ip_allocation_policy bloc note: I fixed these before standing up the cluster, I split out into their own commit for clarity. It was not clear to me how to do a migration for aaa ref: https://www.terraform.io/docs/providers/google/guides/version_3_upgrade.html#automatic-subnetwork-creation-for-vpc-native-clusters-removed
Seeing a lot of "Cloud Logging API has not been used in project 1234 before or it is disabled" on e2e runs
7ff84fb
to
a1d0305
Compare
At the very least, please comment "test-pods" so we know what the heck it
is about and why. I would assume it's some stuff leftover from manual
testing of the cluster
…On Mon, Apr 27, 2020 at 4:22 PM Aaron Crickenberger < ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In infra/gcp/ensure-e2e-projects.sh
<#806 (comment)>:
> + "kubernetes-public" \
+ "${PROW_BUILD_SVCACCT}"
+
+# manual parts:
+# - create key, add to prow-build-test as service-account secret
+# - gsutil iam ch serviceAccount:$PROW_BUILD_SVCACCT:objectAdmin gs://bashfire-prow
+# - gsutil iam ch serviceAccount:$PROW_BUILD_SVCACCT:objectCreator gs://bashfire-prow
+
+# TODO: replace boskos-janitor-test with actual service account
+BOSKOS_JANITOR_SVCACCT=$(svc_acct_email "kubernetes-public" "boskos-janitor-test")
+ensure_service_account \
+ "kubernetes-public" \
+ "boskos-janitor-test" \
+ "used by boskos-janitor in prow-build-test cluster"
+empower_ksa_to_svcacct \
+ "kubernetes-public.svc.id.goog[test-pods/boskos-janitor]" \
comes from a setting in prow:
https://github.com/kubernetes/test-infra/blob/master/config/prow/config.yaml#L73
and the fact that boskos is assumed to run in this specific namespace by
some jobs and/or the same namespace as the jobs that use it
discussed with the prow team and longer term we'd prefer to refactor prow
to support a namespace per build cluster, instead of assuming the same
namespace across all build cluster
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
<#806 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKWAVG3F37YUWZDE6KCMLLROYHTRANCNFSM4MSDSM6A>
.
|
FWIW I consider basically all of this ephemeral, I just wanted to be able to describe how/why I created this prototype. Will sketch out more in #752 as well as discuss during this week's wg-k8s-infra meeting I'm not planning on pointing the actual prow.k8s.io at this until it's not a test cluster |
set -- "${E2E_PROJECTS[@]}" | ||
fi | ||
|
||
for prj; do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The other ensure-* scripts all use color
and indent
to make the output more human-friendly. Any reason not to do that here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It was mostly laziness. Turns out indent didn't work with macOS' default version of sed
, so now we look for gsed
as a fallback. The color 6
thing doesn't actually color output on my terminal but I'm not going to troubleshoot that today.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
/lgtm
/approve
/hold
It wasn't clear if you wanted to commit these "ephemeral" changes or not, so hold - remove at will
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: spiffxp, thockin The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/hold cancel |
Add a kubernetes/prow-build-test cluster provisioned via terraform
Add two service accounts:
Add five gcp projects:
While I was here I needed to script creation of ip addresses so I refactored/added to lib.sh, if need be I can pull those commits into a separate PR
I hooked this build cluster up to my own prow instance so I could iterate on getting things working.
eg: