Can operator-sdk run bundle use local image?

[whg517]

Type of question

Hi, I am writing operator using operator-sdk. Recently I was looking for a solution on how to do e2e testing. I chose to use kind as a temporary test environment.

I started the kind node locally, and then built the operator image and bundle image locally. I used kind load docker-image to load the two images to the kind cluster. When I tried to initialize the test suite environment through operator-sdk run bundle before the test, I found that operator-sdk would pull the images through the network.

In CI environment, these operations are temporary, in order to verify that there is no problem with my logic, I think operator-sdk should first try to use the local bundle image, instead of pulling the remote repository, otherwise you need to provide a test repository to store the temporary image or special version. I think this mechanism is bad for development testing.

I want to ask is this something wrong with me or does operator-sdk run bundle always pull the image of the remote repository?

Question

What did you do?

• make bundle
• make bundle-build
• kind load docker-image quay.io/zncdata/spark-k8s-operator-bundle:vv0.1.0
• operator-sdk run bundle quay.io/zncdata/spark-k8s-operator-bundle:vv0.1.0

output:

INFO[0001] trying next host - response was http.StatusNotFound  host=quay.io
FATA[0001] Failed to run bundle: pull bundle image: error pulling image quay.io/zncdata/spark-k8s-operator-bundle:vv0.1.0: error resolving name for image ref quay.io/zncdata/spark-k8s-operator-bundle:vv0.1.0: quay.io/zncdata/spark-k8s-operator-bundle:vv0.1.0: not found 

What did you expect to see?

operator-sdk shoule use local image builded with docker, rather then pull bundle.

What did you see instead? Under which circumstances?

above

Environment

Operator type:

golang

Kubernetes cluster type:

$ operator-sdk version

operator-sdk version: "v1.33.0", commit: "542966812906456a8d67cf7284fc6410b104e118", kubernetes version: "v1.27.0", go version: "go1.21.5", GOOS: "darwin", GOARCH: "arm64"

$ go version (if language is Go)

go version go1.21.5 darwin/arm64

$ kubectl version

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.4", GitCommit:"f89670c3aa4059d6999cb42e23ccb4f0b9a03979", GitTreeState:"clean", BuildDate:"2023-04-12T12:13:53Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-15T00:38:14Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/arm64"}

Additional context

[whg517]

By reading the source code, I found that the mirror implementation logic supports getting the image from the local, but the local mirror logic is not checked at the time of calling.

operator-sdk/internal/registry/image.go

Lines 30 to 72 in 4abd483

	func ExtractBundleImage(ctx context.Context, logger *log.Entry, image string, local bool, skipTLSVerify bool, useHTTP bool) (string, error) {
	if logger == nil {
	logger = DiscardLogger()
	}
	// Use a temp directory for bundle files. This will likely be removed by
	// the caller.
	wd, err := os.Getwd()
	if err != nil {
	return "", err
	}
	bundleDir, err := os.MkdirTemp(wd, "bundle-")
	if err != nil {
	return "", err
	}
	// This should always work, but if it doesn't bundleDir is still valid.
	if dir, err := filepath.Rel(wd, bundleDir); err == nil {
	bundleDir = dir
	}
	// Export the image into bundleDir.
	logger = logger.WithFields(log.Fields{"dir": bundleDir})
	// Use a containerd registry instead of shelling out to a container tool.
	reg, err := containerdregistry.NewRegistry(
	containerdregistry.WithLog(logger),
	containerdregistry.SkipTLSVerify(skipTLSVerify),
	containerdregistry.WithPlainHTTP(useHTTP))
	if err != nil {
	return "", err
	}
	defer func() {
	if err := reg.Destroy(); err != nil {
	logger.WithError(err).Warn("Error destroying local cache")
	}
	}()
	// Pull the image if it isn't present locally.
	if !local {
	if err := reg.Pull(ctx, registryimage.SimpleReference(image)); err != nil {
	return "", fmt.Errorf("error pulling image %s: %v", image, err)
	}
	}

But at the time of the call, no local judgment is made

operator-sdk/internal/olm/operator/helpers.go

Lines 37 to 40 in 4abd483

	func LoadBundle(ctx context.Context, bundleImage string, skipTLSVerify bool, useHTTP bool) (registryutil.LabelsMap, *apimanifests.Bundle, error) {
	bundlePath, err := registryutil.ExtractBundleImage(ctx, nil, bundleImage, false, skipTLSVerify, useHTTP)
	if err != nil {
	return nil, nil, fmt.Errorf("pull bundle image: %v", err)

I was wondering if anyone could fix this?

[burmanm]

I recommend installing local registry like instructed here: https://kind.sigs.k8s.io/docs/user/local-registry/

After that, you can push to that registry and run a local bundle. For example, my quick test:

➜  cass-operator git:(master) ✗ bin/operator-sdk run bundle localhost:5001/k8ssandra/cass-operator-bundle:v1.20.0-dev.faa4d28-20240404 --use-http
INFO[0008] Creating a File-Based Catalog of the bundle "localhost:5001/k8ssandra/cass-operator-bundle:v1.20.0-dev.faa4d28-20240404" 
INFO[0008] Generated a valid File-Based Catalog         
INFO[0016] Created registry pod: 001-k8ssandra-cass-operator-bundle-v1-20-0-dev-faa4d28-20240404 
INFO[0016] Created CatalogSource: cass-operator-catalog 
INFO[0016] OperatorGroup "operator-sdk-og" created      
INFO[0016] Created Subscription: cass-operator-v1-20-0-dev-faa4d28-20240404-sub 
INFO[0023] Approved InstallPlan install-kbnqd for the Subscription: cass-operator-v1-20-0-dev-faa4d28-20240404-sub 
INFO[0023] Waiting for ClusterServiceVersion "default/cass-operator.v1.20.0-dev.faa4d28-20240404" to reach 'Succeeded' phase 
INFO[0024]   Waiting for ClusterServiceVersion "default/cass-operator.v1.20.0-dev.faa4d28-20240404" to appear 
INFO[0026]   Found ClusterServiceVersion "default/cass-operator.v1.20.0-dev.faa4d28-20240404" phase: Pending 
INFO[0028]   Found ClusterServiceVersion "default/cass-operator.v1.20.0-dev.faa4d28-20240404" phase: Installing 
INFO[0089]   Found ClusterServiceVersion "default/cass-operator.v1.20.0-dev.faa4d28-20240404" phase: Succeeded 
INFO[0089] OLM has successfully installed "cass-operator.v1.20.0-dev.faa4d28-20240404" 
➜  cass-operator git:(master) ✗

Personally I always have local registry installed in my kind setups, since it gives more "e2e" feeling in that sense.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[squakez]

+1 I think this one would be useful. I've tried to setup the local flag, but there are other parts of the execution that are pulling the image later in order to get the bundle into a directory.

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.