(e2e) Use custom built test catalog for e2e testing

[anik120]

This PR

• Stands up a local image registry in the cluster

• Builds bundles and catalog images and uploads them to the image registry

• Uses the custom images in the e2e test suite

• Also introduces a pullSecret field for the Operator API's Spec struct, to allow installation of operators on cluster whose bundles require imagePullSecret to be provisioned. This is required because the bundle images built and uploaded to the local registry above requires a pull secret for the local registry.

closes #215

Description

Reviewer Checklist

• API Go Documentation
• Tests: Unit Tests (and E2E Tests, if appropriate)
• Comprehensive Commit Messages
• Links to related GitHub Issue(s)

[joelanford]

• Stands up a local image registry in the cluster
• Builds bundles and catalog images and uploads them to the image registry

I think this is overkill. We should be able to simply build the images, kind load them, and then refer to them directly in the e2e. No need to run an extra image registry, IMO.

[joelanford]

I think we should emulate the way rukpak set this up:
Build and load here: https://github.com/operator-framework/rukpak/blob/fdc3ebd85c928072fc6f6b59a2debf752159b791/Makefile#L185

Directly refer to those image in e2e's. For example: https://github.com/operator-framework/rukpak/blob/fdc3ebd85c928072fc6f6b59a2debf752159b791/test/e2e/plain_provisioner_test.go#L71

[everettraven]

Stands up a local image registry in the cluster

This feels very heavy handed - is there a reason we have to stand up an image registry on cluster instead of just kind loading the images? As long as we aren't using images with :latest tags the default Kube pull policy should just allow us to use kind load to load images and use them with the unpack pods.

In my head the most simple and straightforward approach to using a custom built test catalog would be to:

1. Build a custom test catalog & pull the bundle image, tagging with a non :latest tag
   2 kind load docker-image ... for both images from 1
2. Change the catalog image in the e2e tests
3. Run the e2e tests

[anik120]

Looks like my use of the latest tag was in fact the problem.

:latest tags the default Kube pull policy

TIL.

Updated the PR to just use built and loaded images

[everettraven]

Overall looks good to me, only have a question about the related images in the bundle blob

[anik120]

Overall looks good to me, only have a question about the related images in the bundle blob

Removed the related images, they aren't needed 👍🏽

[anik120]

/hold

looks like the e2e tests are failing after rebasing on top of #216, have to investigate it

[joelanford]

I'd like to remove this line as well based on my other comment, but I think catalogd may validate the input (and removing this line would cause that validation to fail)

Maybe add a TODO comment to this line that we're only setting this to satisfy the existing OLMv0-based validation logic and that we should look for a way to be able to remove this. WDYT?

[anik120]

Looks like adding comments to a json file isn't that easy.
One way would be to add a key/value of the form "comment": "TODO:...", but that's possibly a bit confusing?

[joelanford]

Oh yeah, forgot about that. Let's just leave it as is.

[anik120]

/hold cancel

[everettraven]

All this seems fine for now. There is likely going to be some modifications as part of #242

[joelanford]

Nit: remove "relatedImages" entirely?

[anik120]

It looks like spec.relatedImages field is not optional:

condition "Unpacked" is False/UnpackFailed; message: "create bundle metadata objects: applying bundle metadata \"test-catalog-prometheusoperator.0.47.0\": BundleMetadata.catalogd.operatorframework.io \"test-catalog-prometheusoperator.0.47.0\" is invalid: spec.relatedImages: Invalid value: \"null\": spec.relatedImages in body must be of type array: \"null\""

Admittedly I'd have preferred to keep it at "relatedImages": [] but even that is read as null, and had to do some json jujitsu to come up with [{}]

[everettraven]

^ Shouldn't be an issue after operator-framework/catalogd#38 is done

[joelanford]

This is a bug in catalogd. We should not require relatedImages to be populated. In fact, I would argue that [{}] is more invalid because it is saying "here's a related image. it's image is "" and it's name is "".

[anik120]

We'll have to create a follow up issue to fix this.

[joelanford]

/approve