Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Managed Identity support for Image Pull #240

Open
wants to merge 116 commits into
base: master
Choose a base branch
from
Open

feat: Managed Identity support for Image Pull #240

Show file tree
Hide file tree
Changes from 115 commits
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
e979e99
ignore helm init and pod specs example
t-ysalazar Jun 7, 2022
bdf2cc1
Revert "ignore helm init and pod specs example"
t-ysalazar Jun 13, 2022
1cc644e
MI for image pull and mock test endpoint
fnuarnav Jun 20, 2022
6264659
add defualt image server when nothing is provided
fnuarnav Jun 20, 2022
a0530aa
added unit tests for MI
fnuarnav Jun 20, 2022
7216493
fix error; remove print statements
fnuarnav Jun 20, 2022
1a4ed8f
removed unused pod identity object
fnuarnav Jun 20, 2022
df701e1
use pointer variable for ContainerGroup.Identity
fnuarnav Jun 20, 2022
d846f64
added comments to describe types
fnuarnav Jun 21, 2022
ec26969
update comments
fnuarnav Jun 21, 2022
14c2f04
only add image reigstry creds for acr servernames
fnuarnav Jun 23, 2022
4f9b1e2
Merge branch 'virtual-kubelet:master' into master
t-ysalazar Jul 7, 2022
3c9d25a
Merge branch 'virtual-kubelet:master' into master
t-ysalazar Jul 11, 2022
b29a7c8
Merge branch 'fnuarnav/feature/mi-for-image-pull' into t-ysalazar/mi-…
t-ysalazar Jul 11, 2022
3b86b2f
rebase with master
t-ysalazar Jul 11, 2022
8adcb62
e2e 1, create cluster
t-ysalazar Jul 12, 2022
c9cda56
E2E delete cluster after test
t-ysalazar Jul 12, 2022
cc98532
e2e get client secret
t-ysalazar Jul 12, 2022
2039759
e2e add managed-identity
t-ysalazar Jul 13, 2022
48fa251
e2e get masterURI
t-ysalazar Jul 13, 2022
8189ffa
e2e helm
t-ysalazar Jul 13, 2022
e2aa4d1
e2e fix miURL
t-ysalazar Jul 14, 2022
e9eb315
e2e fix miURL
t-ysalazar Jul 14, 2022
e890b06
e2e mi pull pod
t-ysalazar Jul 14, 2022
03ca373
TestImagePullUsingKubeletIdentityAndSecrets
t-ysalazar Jul 14, 2022
fc2226e
e2e test TestImagePullUsingKubeletIdentityAndSecrets assign role MI
t-ysalazar Jul 15, 2022
4eabd8e
deployments_test const
t-ysalazar Jul 18, 2022
bfb5706
factorize TestImagePullUsingKubeletIdentity
t-ysalazar Jul 18, 2022
5f441ec
fix delete cluster
t-ysalazar Jul 18, 2022
d69ed08
e2e TestImagePull_KubeletIdentityInAKSCLuster/virtual_node_with_manag…
t-ysalazar Jul 19, 2022
ec1c281
e2e TestAKSDeployment_attachACR
t-ysalazar Jul 19, 2022
38e454f
TestImagePull_KubeletIdentityInAKSCLuster/virtual_node_with_no_secrets
t-ysalazar Jul 20, 2022
fe5cbd1
Merge branch 'virtual-kubelet:master' into master
t-ysalazar Jul 20, 2022
a823e5f
TestImagePull_KubeletIdentityInAKSCLuster
t-ysalazar Jul 20, 2022
884c479
fix merge conflict
t-ysalazar Jul 20, 2022
583d23d
Merge pull request #2 from t-ysalazar/t-ysalazar/mi-for-image-pull
suselva Jul 20, 2022
186a6fb
remove comment
suselva Jul 20, 2022
43df01a
Merge branch 'virtual-kubelet:master' into suselva/mi-image-pull
suselva Jul 21, 2022
d935e7c
update regex string escape
suselva Jul 21, 2022
093ca98
fix compatibility
t-ysalazar Jul 21, 2022
797c58e
parallelization
t-ysalazar Jul 21, 2022
a0ed92a
comments
t-ysalazar Jul 22, 2022
80e7dee
Merge branch 'virtual-kubelet:master' into t-ysalazar/mi-for-image-pull
t-ysalazar Jul 22, 2022
d6693be
Merge pull request #3 from t-ysalazar/t-ysalazar/mi-for-image-pull
t-ysalazar Jul 22, 2022
f9bfbec
Merge branch 't-ysalazar/mi-for-image-pull' of https://github.com/t-y…
t-ysalazar Jul 25, 2022
8545cad
e2e fix node assignation
t-ysalazar Jul 25, 2022
bed4401
Merge pull request #4 from t-ysalazar/t-ysalazar/mi-for-image-pull
t-ysalazar Jul 25, 2022
1ad2fdc
Merge branch 'master' into suselva/mi-image-pull
ryanzhang-oss Jul 27, 2022
c2d3e42
dynamic values in mi-image-pull podspecs
t-ysalazar Aug 4, 2022
f10e459
Merge pull request #5 from t-ysalazar/t-ysalazar/mi-for-image-pull
t-ysalazar Aug 4, 2022
b93be3d
add RG and location to helm; add azure dns ip to aci request
fnuarnav Aug 5, 2022
8a9a647
refractor types.go
suselva Aug 8, 2022
f75d623
acr name may contain numbers
fnuarnav Aug 17, 2022
3826ec7
updated readme for using MI for image pull
fnuarnav Aug 17, 2022
571672c
updated readme
fnuarnav Aug 17, 2022
5c441fd
comment explanation for AzureDnsIp; don't fail when cluster details n…
fnuarnav Aug 17, 2022
3ad9e34
Merge branch 'virtual-kubelet:master' into suselva/mi-image-pull
suselva Aug 19, 2022
0e4734a
lint updates
suselva Aug 23, 2022
d29d133
Merge branch 'master' into suselva/mi-image-pull
suselva Aug 23, 2022
2c14f20
merge changes from master
fnuarnav Aug 30, 2022
965f9f7
merge changes from upstream/master
fnuarnav Sep 6, 2022
3ef7bcf
simplified e2e/deployments_test.go
fnuarnav Sep 6, 2022
a3ce27e
removed unused code
fnuarnav Sep 6, 2022
3f4853b
removed unused import
fnuarnav Sep 6, 2022
a36f2b6
wqMerge branch 'master' into suselva/mi-image-pull
fnuarnav Sep 7, 2022
87a65ab
use correct container name
fnuarnav Sep 7, 2022
1b824ac
use simple command in yaml
fnuarnav Sep 8, 2022
8cde356
ignore yaml created by envsubst
fnuarnav Sep 8, 2022
ff2dcec
merged changes from upstream/master
fnuarnav Sep 9, 2022
9e0e2f6
Merge branch 'master' into suselva/mi-image-pull
fnuarnav Sep 9, 2022
ef82131
Adding print statements to hack script
suselva Sep 9, 2022
ff2a10c
Merge branch 'suselva/mi-image-pull' of https://github.com/suselva/az…
suselva Sep 9, 2022
3805c7b
Fix merge conflict change
suselva Sep 9, 2022
705f196
removed unnecessary comment
fnuarnav Sep 12, 2022
8b19ebf
only create acr in one place
fnuarnav Sep 12, 2022
1f7213d
Merge branch 'master' into suselva/mi-image-pull
fnuarnav Sep 12, 2022
cb28310
merge changes from master
fnuarnav Oct 17, 2022
0a7f3f5
added azidentity and armmsi adk
fnuarnav Oct 28, 2022
4cdd3b1
use managed identity to pull images from ACR
fnuarnav Oct 28, 2022
0926639
added unit test for getImageServerNames method
fnuarnav Oct 28, 2022
8d46c25
added unit tests for getManagedIdentityImageRegistryCredentials
fnuarnav Oct 28, 2022
b6cabe1
removed duplicate test
fnuarnav Oct 31, 2022
1ab605f
Merge remote-tracking branch 'upstream/master' into suselva/mi-image-…
fnuarnav Oct 31, 2022
4179703
added armcontainerservice sdk
fnuarnav Oct 31, 2022
dece820
handle non default resource group and agent pool identity
fnuarnav Oct 31, 2022
cc0c0dc
removed MI from old cold without sdk
fnuarnav Nov 9, 2022
cbfae3a
moved funcs to pkg/provider/identity.go file
fnuarnav Nov 9, 2022
0873ddb
merge with upstream/master
fnuarnav Nov 9, 2022
003b90e
error should start with lowercase
fnuarnav Nov 9, 2022
eeb525d
error should be lowercase
fnuarnav Nov 9, 2022
92e395b
Merge remote-tracking branch 'upstream/master' into suselva/mi-image-…
fnuarnav Nov 10, 2022
3103aa6
cover more cases for finding cluster and kublet identity
fnuarnav Nov 19, 2022
19ed975
merge changes from master
fnuarnav Nov 19, 2022
a31e328
filter by fqdn instead of name since pod.ClusterName is not always pr…
fnuarnav Nov 19, 2022
94de42d
merge changes from master
fnuarnav Jan 3, 2023
25a4c2e
add envsubst statement to aks-addon.sh to use correct acr
fnuarnav Jan 3, 2023
8f6c7bf
add alpine image to the acr to test image pull
fnuarnav Jan 3, 2023
57f5c40
add ManagedIdentityPullFeature to featureflag
fnuarnav Jan 8, 2023
735ac08
skip e2e test if MI feature flag is not set
fnuarnav Jan 8, 2023
7759db4
merge changes from master
fnuarnav Mar 19, 2023
fa28280
updated MI unit tests
fnuarnav Mar 19, 2023
ffd61ed
removed from main
fnuarnav Mar 21, 2023
f37fe30
add alpine image to acr
fnuarnav Mar 21, 2023
e671ca5
Merge branch 'master' into suselva/mi-image-pull
fnuarnav Mar 22, 2023
2b235c4
import ctx
fnuarnav Mar 22, 2023
faecc2c
fix lint error
fnuarnav Mar 22, 2023
89633db
merge changes from master
fnuarnav Mar 30, 2023
64a7e37
merge changes from master
fnuarnav Jul 19, 2023
d8fd3e8
update variable name
fnuarnav Aug 24, 2023
356feeb
update function call
fnuarnav Aug 24, 2023
3d06794
update aks version to 1.26.6 in makefile
fnuarnav Aug 24, 2023
8278fa7
authenticate clients correctly; move readme to docs/; update unit test
fnuarnav Sep 1, 2023
f6624ec
only import image to acr when E2E_TARGET=pr
fnuarnav Sep 1, 2023
34a45ca
fix link in readme
fnuarnav Sep 1, 2023
84bb117
fix lint errors
fnuarnav Sep 1, 2023
e531a87
merge changes from master
fnuarnav Oct 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,9 @@ hack/tools/bin/
.DS_Store
.vscode/
.idea/

.run/
dist/

# File produced during testing
e2e/fixtures/mi-pull-image-exec.yaml
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ OUTPUT_TYPE ?= type=docker
BUILDPLATFORM ?= linux/amd64
IMG_TAG ?= $(subst v,,$(VERSION))
INIT_IMG_TAG ?= 0.2.0
K8S_VERSION ?= 1.24.10
K8S_VERSION ?= 1.26.6
fnuarnav marked this conversation as resolved.
Show resolved Hide resolved

BUILD_DATE ?= $(shell date '+%Y-%m-%dT%H:%M:%S')
VERSION_FLAGS := "-ldflags=-X main.buildVersion=$(IMG_TAG) -X main.buildTime=$(BUILD_DATE)"
Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Virtual Kubelet's ACI provider relies heavily on the feature set that ACI servic
* [Exec support](https://docs.microsoft.com/azure/container-instances/container-instances-exec) for container instances
* Azure Monitor integration ( aka OMS)
* Support for init-containers ([use init containers](#Create-pod-with-init-containers))
* Pull ACR image using managed identity ([acr image pull](./docs/Pull-Images-Using-Managed-Identity.md))

### Limitations (Not supported)

Expand Down Expand Up @@ -184,3 +185,4 @@ If it is an AKS managed virtual node, please follow the steps [here](https://do

<!-- LINKS -->
[az-container-list]: https://docs.microsoft.com/cli/azure/container?view=azure-cli-latest#az_container_list

28 changes: 28 additions & 0 deletions docs/Pull-Images-Using-Managed-Identity.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
### Pulling Images Using AKS Managed identity
If your image is on a private reigstry, the AKS agent pool identity can be used to pull the images

Attach the private acr registry to the cluster. This will give AcrPull access to the AKS agent pool managed identity
```bash
az aks update -g <resource group> -n <cluster name> --attach-acr <acr name>
```

Create a new pod that pulls an image from the private registry, for example
```yaml
spec:
containers:
- image: <ACR NAME>.azurecr.io/<IMAGE NAME>:<IMAGE TAG>
name: test-container
```

#### Optional: Use a Custom Managed Identity
To use a custom manged identity instead of the AKS agent pool identity, it must be added as a kubelet identity on the aks cluster.
```bash
az identity create -g <RESOURCE GROUP> -n <USER ASSIGNED IDENTITY NAME>
az aks update -g <RESOURCE GROUP> -n <CLUSER NAME> --assign-kubelet-identity <USER ASSIGNED IDENTITY URI>
```

Then Attach the container registry
```bash
az aks update -g <resource group> -n <cluster name> --attach-acr <acr name>
```

90 changes: 90 additions & 0 deletions e2e/deployments_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
package e2e

import (
"testing"
"time"
"context"

"github.com/virtual-kubelet/azure-aci/pkg/featureflag"
)

func TestImagePullUsingKubeletIdentityMI(t *testing.T) {
ctx := context.TODO()
enabledFeatures := featureflag.InitFeatureFlag(ctx)
if !enabledFeatures.IsEnabled(ctx, featureflag.ManagedIdentityPullFeature) {
t.Skipf("%s feature is not enabled", featureflag.ManagedIdentityPullFeature)
}
// delete the pod first
cmd := kubectl("delete", "namespace", "vk-test", "--ignore-not-found")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatal(string(out))
}

// create namespace
cmd = kubectl("apply", "-f", "fixtures/namespace.yml")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatal(string(out))
}

// run container group pulling image from acr using MI
cmd = kubectl("apply", "-f", "fixtures/mi-pull-image-exec.yaml")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatal(string(out))
}

deadline, ok := t.Deadline()
timeout := time.Until(deadline)
if !ok {
timeout = 300 * time.Second
}
cmd = kubectl("wait", "--for=condition=ready", "--timeout="+timeout.String(), "pod/e2etest-acr-test-mi-container", "--namespace=vk-test")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatal(string(out))
}
t.Log("success pulling image from ACR using managed identity")

// query metrics
deadline = time.Now().Add(5 * time.Minute)
for {
t.Log("query metrics ....")
cmd = kubectl("get", "--raw", "/apis/metrics.k8s.io/v1beta1/namespaces/vk-test/pods/e2etest-acr-test-mi-container")
out, err := cmd.CombinedOutput()
if time.Now().After(deadline) {
t.Fatal("failed to query pod's stats from metrics server API")
}
if err == nil {
t.Logf("success query metrics %s", string(out))
break
}
}

// check pod status
t.Log("get pod status ....")
cmd = kubectl("get", "pod", "--field-selector=status.phase=Running", "--namespace=vk-test", "--output=jsonpath={.items..metadata.name}")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatal(string(out))
}
if string(out) != "e2etest-acr-test-mi-container" {
t.Fatal("failed to get pod's status")
}
t.Logf("success query pod status %s", string(out))

// check container status
t.Log("get container status ....")
cmd = kubectl("get", "pod", "e2etest-acr-test-mi-container", "--namespace=vk-test", "--output=jsonpath={.status.containerStatuses[0].ready}")
out, err = cmd.CombinedOutput()
if err != nil {
t.Fatal(string(out))
}
if string(out) != "true" {
t.Fatal("failed to get pod's status")
}
t.Logf("success query container status %s", string(out))

t.Log("clean up pod")
cmd = kubectl("delete", "namespace", "vk-test", "--ignore-not-found")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatal(string(out))
}
}
28 changes: 28 additions & 0 deletions e2e/fixtures/mi-pull-image.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
apiVersion: v1
kind: Pod
metadata:
name: e2etest-acr-test-mi-container
namespace: vk-test
spec:
nodeName: vk-aci-test-aks
restartPolicy: Never
containers:
- image: ${ACR_NAME}.azurecr.io/library/alpine
imagePullPolicy: Always
name: e2etest-acr-test-mi-container
command: [
"sh",
"-c",
"sleep 1; while sleep 1; do echo pulled image using mi; done",
]
resources:
requests:
memory: 1G
cpu: 1
nodeSelector:
kubernetes.io/role: agent
beta.kubernetes.io/os: linux
type: virtual-kubelet
tolerations:
- key: virtual-kubelet.io/provider
operator: Exists
1 change: 1 addition & 0 deletions e2e/fixtures_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"os/exec"
)

//execute kubectl command in terminal
func kubectl(args ...string) *exec.Cmd {
cmd := exec.Command("kubectl", args...)
cmd.Env = os.Environ()
Expand Down
16 changes: 9 additions & 7 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@ go 1.20

require (
contrib.go.opencensus.io/exporter/ocagent v0.7.0
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.1.4
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.0
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.4.0
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.2
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerinstance/armcontainerinstance/v2 v2.2.0-beta.1
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice v1.0.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi v1.0.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v2 v2.1.0
github.com/Azure/go-autorest/autorest v0.11.27
github.com/Azure/go-autorest/autorest/adal v0.9.20
Expand Down Expand Up @@ -36,12 +38,12 @@ require (
)

require (
github.com/Azure/azure-sdk-for-go/sdk/internal v1.0.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.2.0 // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v0.7.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v0.9.0 // indirect
github.com/NYTimes/gziphandler v1.1.1 // indirect
github.com/antlr/antlr4/runtime/Go/antlr v1.4.10 // indirect
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a // indirect
Expand All @@ -63,7 +65,7 @@ require (
github.com/go-openapi/jsonreference v0.20.1 // indirect
github.com/go-openapi/swag v0.22.3 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.4.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/google/cel-go v0.12.6 // indirect
Expand All @@ -85,7 +87,7 @@ require (
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_golang v1.15.1 // indirect
github.com/prometheus/common v0.42.0 // indirect
Expand All @@ -108,7 +110,7 @@ require (
go.uber.org/atomic v1.7.0 // indirect
go.uber.org/multierr v1.6.0 // indirect
go.uber.org/zap v1.24.0 // indirect
golang.org/x/crypto v0.1.0 // indirect
golang.org/x/crypto v0.6.0 // indirect
golang.org/x/net v0.10.0 // indirect
golang.org/x/oauth2 v0.8.0 // indirect
golang.org/x/sys v0.9.0 // indirect
Expand Down
33 changes: 19 additions & 14 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,15 +47,19 @@ cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9
contrib.go.opencensus.io/exporter/ocagent v0.7.0 h1:BEfdCTXfMV30tLZD8c9n64V/tIZX5+9sXiuFLnrr1k8=
contrib.go.opencensus.io/exporter/ocagent v0.7.0/go.mod h1:IshRmMJBhDfFj5Y67nVhMYTTIze91RUeT73ipWKs/GY=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.1.4 h1:pqrAR74b6EoR4kcxF7L7Wg2B8Jgil9UUZtMvxhEFqWo=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.1.4/go.mod h1:uGG2W01BaETf0Ozp+QxxKJdMBNRWPdstHG0Fmdwn1/U=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.0 h1:t/W5MYAuQy81cvM8VUNfRLzhtKpXhVUAN7Cd7KVbTyc=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.0/go.mod h1:NBanQUfSWiWn3QEpWDTCU0IjBECKOYvl2R8xdRtMtiM=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.0.0 h1:jp0dGvZ7ZK0mgqnTSClMxa5xuRL7NZgHameVYF6BurY=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.0.0/go.mod h1:eWRD7oawr1Mu1sLCawqVc0CUiF43ia3qQMxLscsKQ9w=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.4.0 h1:rTnT/Jrcm+figWlYz4Ixzt0SJVR2cMC8lvZcimipiEY=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.4.0/go.mod h1:ON4tFdPTwRcgWEaVDrN3584Ef+b7GgSJaXxe5fW9t4M=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.2 h1:uqM+VoHjVH6zdlkLF2b6O0ZANcHoj3rO0PoQ3jglUJA=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.2/go.mod h1:twTKAa1E6hLmSDjLhaCkbTMQKc7p/rNLU40rLxGEOCI=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.2.0 h1:leh5DwKv6Ihwi+h60uHtn6UWAxBbZ0q8DwQVMzf61zw=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.2.0/go.mod h1:eWRD7oawr1Mu1sLCawqVc0CUiF43ia3qQMxLscsKQ9w=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerinstance/armcontainerinstance/v2 v2.2.0-beta.1 h1:eY6fhA944YceJrJ9OGn1T5iqe5DA2rQ+O1/Gi3P4bXU=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerinstance/armcontainerinstance/v2 v2.2.0-beta.1/go.mod h1:5Q/hN8CkM0y7bBldgIdoPMp9jyBZ1KVeexQvfY2KXw8=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice v1.0.0 h1:figxyQZXzZQIcP3njhC68bYUiTw45J8/SsHaLW8Ax0M=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice v1.0.0/go.mod h1:TmlMW4W5OvXOmOyKNnor8nlMMiO1ctIyzmHme/VHsrA=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal v1.0.0 h1:lMW1lD/17LUA5z1XTURo7LcVG2ICBPlyMHjIUrcFZNQ=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi v1.0.0 h1:ZOt3s8LxEoRGgdD/k7Co4wGAWKmO4+jdPRCRBa8Rzc0=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi v1.0.0/go.mod h1:ZJWUTTEMZLTJI4PPI6vuv/OCEs9YjEX9EqjCnLJ8afA=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v2 v2.1.0 h1:mk57wRUA8fyjFxVcPPGv4shLcWDXPFYokTJL9zJxQtE=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v2 v2.1.0/go.mod h1:mU96hbp8qJDA9OzTV1Ji7wCyPyaqC5kI6ZPsZfJ8sE4=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.0.0 h1:ECsQtyERDVz3NP3kvDOTLvbQhqWp/x9EsGKtb4ogUr8=
Expand All @@ -75,8 +79,8 @@ github.com/Azure/go-autorest/logger v0.2.1 h1:IG7i4p/mDa2Ce4TRyAO8IHnVhAVF3RFU+Z
github.com/Azure/go-autorest/logger v0.2.1/go.mod h1:T9E3cAhj2VqvPOtCYAvby9aBXkZmbF5NWuPV8+WeEW8=
github.com/Azure/go-autorest/tracing v0.6.0 h1:TYi4+3m5t6K48TGI9AUdb+IzbnSxvnvUMfuitfgcfuo=
github.com/Azure/go-autorest/tracing v0.6.0/go.mod h1:+vhtPC754Xsa23ID7GlGsrdKBpUA79WCAKPPZVC2DeU=
github.com/AzureAD/microsoft-authentication-library-for-go v0.7.0 h1:VgSJlZH5u0k2qxSpqyghcFQKmvYckj46uymKK5XzkBM=
github.com/AzureAD/microsoft-authentication-library-for-go v0.7.0/go.mod h1:BDJ5qMFKx9DugEg3+uQSDCdbYPr5s9vBTrL9P8TpqOU=
github.com/AzureAD/microsoft-authentication-library-for-go v0.9.0 h1:UE9n9rkJF62ArLb1F3DEjRt8O3jLwMWdSoypKV4f3MU=
github.com/AzureAD/microsoft-authentication-library-for-go v0.9.0/go.mod h1:kgDmCTgBzIEPFElEF+FK0SdjAor06dRq2Go927dnQ6o=
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
Expand Down Expand Up @@ -167,8 +171,8 @@ github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt/v4 v4.0.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg=
github.com/golang-jwt/jwt/v4 v4.2.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg=
github.com/golang-jwt/jwt/v4 v4.4.2 h1:rcc4lwaZgFMCZ5jxF9ABolDcIHdBytAFgqFPbSJQAYs=
github.com/golang-jwt/jwt/v4 v4.4.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg=
github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/glog v1.1.0 h1:/d3pCKDPWNnvIWe0vVUpNP32qc8U3PDVxySP/y360qE=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
Expand Down Expand Up @@ -316,8 +320,8 @@ github.com/onsi/ginkgo/v2 v2.9.1 h1:zie5Ly042PD3bsCvsSOPvRnFwyo3rKe64TJlD6nu0mk=
github.com/onsi/gomega v1.27.4 h1:Z2AnStgsdSayCMDiCU42qIz+HLqEPcgiOCXjAU/w+8E=
github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 h1:Qj1ukM4GlMWXNdMBuXcXfz/Kw9s1qm0CLY32QxuSImI=
github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4/go.mod h1:N6UoU20jOqggOuDwUaBQpluzLNDqif3kq9z2wpdYEfQ=
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 h1:KoWmjvw+nsYOo29YJK9vDA65RGE3NrOnUtO7a+RF9HU=
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8/go.mod h1:HKlIX3XHQyzLZPlr7++PzdhaXEj94dEiJgZDTsxEqUI=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
Expand Down Expand Up @@ -422,8 +426,8 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.1.0 h1:MDRAIl0xIo9Io2xV565hzXHw3zVseKrJKodhohM5CjU=
golang.org/x/crypto v0.1.0/go.mod h1:RecgLatLF4+eUMCP1PoPZQb+cVrJcOPbHkTkbkB9sbw=
golang.org/x/crypto v0.6.0 h1:qfktjS5LUO+fFKeJXZ+ikTRijMmljikvG68fpMMruSc=
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
Expand Down Expand Up @@ -569,6 +573,7 @@ golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210603125802-9665404d3644/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616045830-e2b7044e8c71/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
Expand Down
11 changes: 8 additions & 3 deletions hack/e2e/aks-addon.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,12 @@ if [ "$E2E_TARGET" = "pr" ]; then
IMG_URL=$ACR_NAME.azurecr.io
OUTPUT_TYPE=type=registry IMG_TAG=$IMG_TAG IMAGE=$IMG_URL/$IMG_REPO make docker-build-image
OUTPUT_TYPE=type=registry INIT_IMG_TAG=$INIT_IMG_TAG INIT_IMAGE=$IMG_URL/$INIT_IMG_REPO make docker-build-init-image

az acr import --name ${ACR_NAME} --source docker.io/library/alpine:latest
fi

export ACR_ID="$(az acr show --resource-group ${RESOURCE_GROUP} --name ${ACR_NAME} --query id -o tsv)"
export ACR_NAME=${ACR_NAME}

TMPDIR="$(mktemp -d)"

az network vnet create \
Expand Down Expand Up @@ -161,14 +164,14 @@ kubectl create configmap test-vars -n kube-system \

sed -e "s|TEST_INIT_IMAGE|$IMG_URL/$INIT_IMG_REPO:$INIT_IMG_TAG|g" -e "s|TEST_IMAGE|$IMG_URL/$IMG_REPO:$IMG_TAG|g" deploy/deployment.yaml | kubectl apply -n kube-system -f -

kubectl wait --for=condition=available deploy "virtual-kubelet-azure-aci" -n kube-system --timeout=300s
kubectl wait --for=condition=available deploy "virtual-kubelet-azure-aci" -n kube-system --timeout=600s

while true; do
kubectl get node "$TEST_NODE_NAME" &> /dev/null && break
sleep 3
done

kubectl wait --for=condition=Ready --timeout=300s node "$TEST_NODE_NAME"
kubectl wait --for=condition=Ready --timeout=600s node "$TEST_NODE_NAME"

export TEST_NODE_NAME

Expand Down Expand Up @@ -209,4 +212,6 @@ CSI_DRIVER_STORAGE_ACCOUNT_KEY=$(az storage account keys list --resource-group "
export CSI_DRIVER_STORAGE_ACCOUNT_NAME=$CSI_DRIVER_STORAGE_ACCOUNT_NAME
export CSI_DRIVER_STORAGE_ACCOUNT_KEY=$CSI_DRIVER_STORAGE_ACCOUNT_KEY

envsubst < e2e/fixtures/mi-pull-image.yaml > e2e/fixtures/mi-pull-image-exec.yaml

$@
Loading
Loading