Ansible Operator Plugin Extraction

[everettraven]

Motivation

The extraction of the Ansible Operator Plugin (and other associated items) is meant to help simplify the maintenance burden of the Operator-SDK by only keeping the components that are essential to the core Operator-SDK CLI in the Operator-SDK repository.

By extracting the Ansible Operator Plugin from the Operator-SDK repository and moving it to it's own repository, we ensure that we can iterate and make changes to the Ansible Operator Plugin's master branch without causing the Operator-SDK CI and/or releases to break.

Goals

• The Ansible Operator Plugin and it's related components no longer live in https://github.com/operator-framework/operator-sdk and instead live in it's own repository under the Operator Framework GitHub Organization (maybe operator-framework/ansible-operator-plugin?)

How can we accomplish this?

After some discussion in the community, there are a few open questions that this section will provide potential solutions for:

• What code do we actually need to move?
• How can we test the Ansible Operator Plugin?
• How are we going to version and release the Ansible Operator Plugin?

Component Extraction

At a high level, the following would need to be extracted:

• Ansible Operator binary
• Ansible Operator images
• Ansible Operator Plugin
• Ansible Operator Controller code

At a lower level, the following sections of code would likely need to be moved:

• Binary:
  • https://github.com/operator-framework/operator-sdk/tree/master/cmd/ansible-operator
  • https://github.com/operator-framework/operator-sdk/tree/master/internal/cmd/ansible-operator
• Images:
  • https://github.com/operator-framework/operator-sdk/tree/master/images/ansible-operator
  • https://github.com/operator-framework/operator-sdk/tree/master/images/ansible-operator-2.11-preview
• Plugin:
  • https://github.com/operator-framework/operator-sdk/tree/master/internal/plugins/ansible/v1
• Controller:
  • https://github.com/operator-framework/operator-sdk/tree/master/internal/ansible

We would also need to remove from the Operator-SDK:

• Testdata generation:
  • https://github.com/operator-framework/operator-sdk/tree/master/hack/generate/samples/molecule
  • https://github.com/operator-framework/operator-sdk/tree/master/hack/generate/samples/internal/ansible
• Testdata:
  • https://github.com/operator-framework/operator-sdk/tree/master/testdata/ansible/memcached-operator
• Ansible related CI:
  • https://github.com/operator-framework/operator-sdk/blob/master/.github/workflows/test-ansible.yml
  • https://github.com/operator-framework/operator-sdk/blob/master/.github/workflows/deploy-manual.yml
• Makefile targets:

  • operator-sdk/Makefile

    Lines 98 to 101 in 9d2f672

    	# Convenience wrapper for building dependency base images.
    	.PHONY: image-build-base
    	IMAGE_BASE_TARGET_LIST = ansible-operator ansible-operator-2.11-preview
    	image-build-base: $(foreach i,$(IMAGE_BASE_TARGET_LIST),image-base/$(i)) ## Build all images.

  • operator-sdk/Makefile

    Lines 196 to 201 in 9d2f672

    	test-e2e-ansible:: image/ansible-operator ## Run Ansible e2e tests
    	go test -count=1 ./internal/ansible/proxy/...
    	go test ./test/e2e/ansible -v -ginkgo.v
    	test-e2e-ansible-molecule:: install dev-install image/ansible-operator ## Run molecule-based Ansible e2e tests
    	go run ./hack/generate/samples/molecule/generate.go
    	./hack/tests/e2e-ansible-molecule.sh

Note: We shouldn't remove anything from the main Operator-SDK repository until the extraction of the Ansible Operator Plugin is completed.

Testing

An important part of extracting this plugin is ensuring that we are still able to test the plugin and the associated components. Unit testing things like the Ansible Operator binary and the controller implementations should be relatively easy as it should be a direct port.

E2E testing is where it gets a little trickier but I think can be done in a couple different ways.

Building the SDK from source, substituting the plugin dependency

One way in which we can e2e test the plugin is to clone the Operator-SDK source, substitute the plugin dependency with a local version in the go.mod, build the Operator-SDK and then continue on testing the same way the Operator-SDK currently does.

To substitute the plugin dependency we would need to inject a dependency replacement in the go.mod that looks like:

replaces (
    github.com/operator-framework/ansible-operator-plugin@v1 => /path/to/local/ansible-operator-plugin
)

This process is a fairly complex e2e workflow, but ensures that the ansible operator plugin changes will work with the current changes in the Operator-SDK repository.

Using Kubebuilder CLI library

Since the Operator-SDK utilizes the the Kubebuilder CLI library to actually handle the scaffolding logic and plugins, we could use the same Kubebuilder CLI library to create our testdata. While this means it doesn't get built into the Operator-SDK binary for testing, it would be following a very similar (if not identical) flow. This would keep the e2e workflow much more simple and entirely contained to the ansible operator plugin repository.

This would mean implementing a testdata generation script that creates and runs the testdata CLI. This could look something like:

// Keeping this very simple, but for higher confidence in our testing we
// could ensure the bundle uses all the same plugins the Operator-SDK uses
ansibleBundle, _ := plugin.NewBundle("ansible"+plugins.DefaultNameQualifier, plugin.Version{Number: 1},
	ansiblev1.Plugin{},
)

c, err := cli.New(
	cli.WithCommandName("ansible-plugin-tester"),
	cli.WithVersion("v1"),
	cli.WithPlugins(
		ansibleBundle,
	),
	cli.WithDefaultPlugins(cfgv3.Version, ansibleBundle),
	cli.WithDefaultProjectVersion(cfgv3.Version),
)

if err != nil {
	log.Fatal(err)
}

This way we have a minimal CLI that closely mimics the Operator-SDK used specifically for our testdata generation. Once the testdata is generated, we can follow the same process that is used by the Operator-SDK for it's current e2e testing.

Versioning and Releasing

Moving the Ansible Operator Plugin to it's own repository would mean we need to release and version this repository separately from the Operator-SDK.

The majority of the effort would be around creating the release process, which would likely require writing automation and release scripts to create a release based on a pushed tag. For a particular release version the following artifacts should be produced:

• Ansible Operator binary
• Ansible Operator images

A potential versioning strategy for this separate repository would be to start the first release after extraction at v1.0.0 (the current version of the ansible plugin) and follow semver for future releases. Ideally there are no changes made to the core of the ansible operator plugin during the extraction process and a v1.0.0 release is cut immediately after the extraction is finalized.

[jberkhahn]

Would it be a good idea to dogfood v3 plugins via this?. We said in the past that that would require a major version bump of sdk and distributing multiple binaries, are we sure that is the case? Is it a worth doing even if that's true?

[everettraven]

@jberkhahn By v3 plugins do you mean the Phase 2 Plugins introduced by Kubebuilder? (these are the external plugins that require a binary and would therefore not be shipped as part of the SDK binary)

[varshaprasad96]

Moving ansible plugin out of SDK seems a great step towards making the development and release process of the plugin and the SDK binary easier. Doing so using phase 1.5 plugin architecture does bring in some of the questions which I'm adding below. It would be helpful to know if these have already been discussed and investigated before proceeding:

1. Creating a framework to be able to do an e2e test of the ansible operator as done currently in SDK.
   This includes testing with OLM, as well as bundling with other plugins like manifests/v2 and scorecard which is done currently. These plugins are internal in SDK currently and hence cannot be tested using the KB cli library. A very hacky duck-tape mechanism to build SDK binary locally was implemented in helm-operator plugins (Framework for testing helm and hybrid plugins in this repository helm-operator-plugins#104), but I am extremely hesitant with making the same mistake again. We could just test the ansible scaffolding with kustomize, but would still be running against the fear of the changes not being compatible with other internal plugins.

2. Discussion on maintaining the release cycle of ansible plugins in SDK? Would there be separate releases of ansible binary as well as the one which is bundled with other plugins? What is the value add in doing so? What plugins would the ansible operator binary (as mentioned in the discussion) contain?

3. A long punted question of maintaining documentation for plugins (could be an easier decision to make - maybe, just that we need to make one). Where would the documentation for specific plugins be located. If we have a plugin with different version, and SDK binary with a different one, how would the release branch doc look like? This concerns even the phase 2 implementation.

4. I'm sure its not relevant to ansible, but adding it in for the purposes of future plugins. If the plugin scaffolding, has a dependency on the library that is present in the same go module and needs to be imported, it becomes difficult testing the scaffolding with the latest commit (for ex. in here). Using replaces or go work is hacky but not a long term solution.

I'm not against externalizing plugins and completely understand the benefits of doing so, which is the whole reason why phase 2 was introduced. If phase 2 intends to solve the issues we are facing now, and is something that we encourage other plugin authors to follow, why shouldn't we ourselves do the same? Though I understand that this would require creating separate binaries which are not a part of SDK, and we are hesitant making that change in a minor version, we could still work towards it by externalizing using the phase 2 plugins first, followed by removing them internally from SDK. The SDK users would still not face any breaking change in terms of plugin scaffoldings with phase 2 - which is the most important factor (if not, it would have been reasonable to do SDK 2.0).

Is making a bunch of changes to SDK currently to externalize the plugin with phase 1.5 and carrying over the tech debt of moving to phase 2.0 really worth it? Why can't we put in the efforts for phase 2 right away and not punt the decision for later.

[jberkhahn]

er yeah, v2 plugins. I forget where we are with the numbers

[everettraven]

In the community meeting we discussed breaking this out into a Phase 2 Plugin. I'm going to be creating a new repository under my GitHub account to extract all the ansible specific logic and begin the process.

This is likely to be a long running effort so I will post updates here as I have time and will do my best to include what I think the next steps would be in each update. Thanks!

[everettraven]

My stab at extracting the ansible specific logic: https://github.com/everettraven/ansible-external-poc

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[everettraven]

/remove-lifecycle stale

/lifecycle frozen