Add infra scripts/terraform for prow build cluster prototype #806
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
cncf-cla: yes
k8s-ci-robot
added
wg/k8s-infra
approved
spiffxp
force-pushed
the
prow-build-test
branch
from
2984b9c
to
7ff84fb
Compare
spiffxp
added
area/infra
|
/uncc @mikedanese |
k8s-ci-robot
requested review from
bartsmykla and
dims
and removed request for
mikedanese
|
I have already actuated the stuff in this PR, so I'm running audit now to open a separate PR catching these changes (and anything else that may have happened) |
|
#807 is the audit |
thockin
reviewed
Apr 27, 2020
specifically: - copied aaa and renamed - followed IMPORTANT comments for test cluster EXCEPT - used n1-highmem-2 because e2e jobs eat memory like candy and were getting evicted from n1-standard-1's
address terraform plan warnings, so: - no needless interpolation - got rid of ip_allocation_policy bloc note: I fixed these before standing up the cluster, I split out into their own commit for clarity. It was not clear to me how to do a migration for aaa ref: https://www.terraform.io/docs/providers/google/guides/version_3_upgrade.html#automatic-subnetwork-creation-for-vpc-native-clusters-removed
Seeing a lot of "Cloud Logging API has not been used in project 1234 before or it is disabled" on e2e runs
spiffxp
force-pushed
the
prow-build-test
branch
from
7ff84fb
to
a1d0305
Compare
|
At the very least, please comment "test-pods" so we know what the heck it
is about and why. I would assume it's some stuff leftover from manual
testing of the cluster
…On Mon, Apr 27, 2020 at 4:22 PM Aaron Crickenberger < ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In infra/gcp/ensure-e2e-projects.sh
<#806 (comment)>:
> + "kubernetes-public" \
+ "${PROW_BUILD_SVCACCT}"
+
+# manual parts:
+# - create key, add to prow-build-test as service-account secret
+# - gsutil iam ch serviceAccount:$PROW_BUILD_SVCACCT:objectAdmin gs://bashfire-prow
+# - gsutil iam ch serviceAccount:$PROW_BUILD_SVCACCT:objectCreator gs://bashfire-prow
+
+# TODO: replace boskos-janitor-test with actual service account
+BOSKOS_JANITOR_SVCACCT=$(svc_acct_email "kubernetes-public" "boskos-janitor-test")
+ensure_service_account \
+ "kubernetes-public" \
+ "boskos-janitor-test" \
+ "used by boskos-janitor in prow-build-test cluster"
+empower_ksa_to_svcacct \
+ "kubernetes-public.svc.id.goog[test-pods/boskos-janitor]" \
comes from a setting in prow:
https://github.com/kubernetes/test-infra/blob/master/config/prow/config.yaml#L73
and the fact that boskos is assumed to run in this specific namespace by
some jobs and/or the same namespace as the jobs that use it
discussed with the prow team and longer term we'd prefer to refactor prow
to support a namespace per build cluster, instead of assuming the same
namespace across all build cluster
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
<#806 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKWAVG3F37YUWZDE6KCMLLROYHTRANCNFSM4MSDSM6A>
.
|
|
FWIW I consider basically all of this ephemeral, I just wanted to be able to describe how/why I created this prototype. Will sketch out more in #752 as well as discuss during this week's wg-k8s-infra meeting I'm not planning on pointing the actual prow.k8s.io at this until it's not a test cluster |
thockin
reviewed
Apr 28, 2020
| set -- "${E2E_PROJECTS[@]}" | ||
| fi | ||
|
|
||
| for prj; do |
There was a problem hiding this comment.
The other ensure-* scripts all use color and indent to make the output more human-friendly. Any reason not to do that here?
There was a problem hiding this comment.
It was mostly laziness. Turns out indent didn't work with macOS' default version of sed, so now we look for gsed as a fallback. The color 6 thing doesn't actually color output on my terminal but I'm not going to troubleshoot that today.
thockin
reviewed
Apr 28, 2020
k8s-ci-robot
added
the
do-not-merge/hold
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: spiffxp, thockin The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/hold cancel |
k8s-ci-robot
removed
the
do-not-merge/hold
Merged
spiffxp
added
the
area/prow
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a kubernetes/prow-build-test cluster provisioned via terraform
Add two service accounts:
Add five gcp projects:
While I was here I needed to script creation of ip addresses so I refactored/added to lib.sh, if need be I can pull those commits into a separate PR
I hooked this build cluster up to my own prow instance so I could iterate on getting things working.
eg: