Bootstrap cluster with ignition

[mkumatag]

/kind feature
/area provider/ibmcloud

Describe the solution you'd like
https://cluster-api.sigs.k8s.io/tasks/experimental-features/ignition.html

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[mkumatag]

/remove-lifecycle stale

[mkumatag]

Fedora CoreOS is still pending for ppc64le, will explore this task once builds are available for the testing.

[Prajyot-Parab]

/assign @Prajyot-Parab
Update - Explore on booting Fedora CoreOS with ignition for CAPIBM PowerVS K8s cluster.

@mkumatag can you please add the reference links/urls for the Fedora CoreOS (ppc64le) builds.

[yussufsh]

@Prajyot-Parab https://fedoraproject.org/coreos/download/?stream=stable&arch=ppc64le#download_section you can download ppc64le stable images from here.

[mkumatag]

@Prajyot-Parab can you document the initial investigation you have done and move this work item to next release.

[Prajyot-Parab]

/milestone Next

[Prajyot-Parab]

The default configuration engine for bootstrapping workload cluster machines is cloud-init. Ignition is an alternative engine used by Linux distributions such as Flatcar Container Linux and Fedora CoreOS and therefore should be used when choosing an Ignition-based distribution as the underlying OS for workload clusters.

Note
This initial implementation uses Ignition v2 and was tested with Flatcar Container Linux only. Future releases are expected to add Ignition v3 support and cover more Linux distributions.

@mkumatag as per my initial understanding we will need to build the flatcar stable image for powervs, once we have that we can try the already tested workflow using it.

Not sure how this is going to work with Fedora CoreOS-

1. We will need some modifications on top of it to make it CAPIBM ready right? If yes, will this be somehow done as part of image builder process or it will be dealt with externally?

@mkumatag need your guidance on this one.

[mkumatag]

Flatcar image is not available for the power, let's explore the experiment with the fedoracore image directly and see what kind of errors we are hitting while running the flow?!

[Prajyot-Parab]

Summary -
We explored with FerdoraCoreOS 39 for Power

1. The version of ignition inside image is 2.17.0 => 3.x.x spec
2. The created infra machine was able to fetch the ignition from mentioned URL.
3. The issue is with config version mismatch while booting above machine via ignition. This is result of ignition file being generated using older spec 2.3.0.
4. Ignition file is generated while upstream CAPI (which in-turn uses flatcar package), which supports only 2.3.0 and no 3.x.x specs.

This will possibly need more exploration and upstream CAPI contributions to add support for spec 3.x.x ignition versions.

[Prajyot-Parab]

@Karthik-K-N feel free to add more info, if i missed anything.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[mkumatag]

/remove-lifecycle stale

[Karthik-K-N]

WIP PR is submitted #1687, Need to complete it.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Karthik-K-N]

Currently CAPIBM controllers supports booting machine with ignition with 3.x version which is extensively used in Openshift clusters where the ignition is generated by installer and which has version 3.x.

As a standalone CAPIBM cluster we are yet to support ignition because the CAPI has support for generating ignition with only 2.3 version.

Possible goals?

1. Should we make changes in CAPIBM to support very old ignition 2.3.X?
   (is it worth to make changes as ignition 3.x flow will always be tested in IPI flow)

Challenges

1. Many other providers use flatcar which supports ignition and image builder repo also has support for building flatcar image for CAPI, Unfortunately it does not support ppc64le arch.

Alternatives

1. Use very old rchcos image which has igntion 2.3.x support, make changes in image builder to build CAPI compatible image for this and try with CAPIBM.
2. Identify and use converter package which converts ignition from 2.3.x to 3.x

Upstream CAPI issues where its currently in planning state to support ignition 3.x version.

1. Ignition v3.x support cluster-api#9157
2. Cleaner separation of kubeadm and machine bootstrapping cluster-api#5294

[mkumatag]

@Karthik-K-N can we list down the pro and cons for both the approaches mentioned and debate from there?