Ignition v3.x support

[bengentil]

What would you like to be added (User Story)?

As a user I would like to bootstrap Flatcar & FCOS with Ignition v3.x as Ignition v2.x doesn't work with FCOS and is only required for flatcar LTS release.

Detailed Description

The current implementation (Container Linux Config aka clc) is used when setting format to "ignition" in KubeadmConfigSpec.
To be able to support both v2.x & v3.x, a new field needs to be added to know which transpiler should be used:

• clc for v2.x
• butane for v3.x

Like in the current implementation, an additional butane config may be provided to extend/override the generated Ignition config.

Anything else you would like to add?

This issue follows this slack conversation: https://kubernetes.slack.com/archives/C8TSNPY4T/p1688643597808979

Label(s) to be applied

/kind feature
/area bootstrap
/area provider/bootstrap-kubeadm

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vincepri]

/priority important-soon

[JoelSpeed]

I've been looking into this a little and concluded that resolving this should be relatively straight forward, at least, on the providers side.

Looking at AWS as an example. The only thing it does for ignition is to upload the existing ignition data to s3 and then create a stub ignition which points to the s3 bucket.

There are minor differences between v2 and v3, but, it's not hard to create a switch and support this by importing both the v2 and v3 types.

For kubeadm this may be a little bit more involved.

Does anyone know about the flatcar ignition fork vs the coreos ignition? I was looking at both today and as far as I can tell they are API compatible, but the coreos fork seems to be more active at the moment?

[bengentil]

The PR addressing this issue is currently waiting on progress with #5294

see #9158 (comment) for more details

[johananl]

Does anyone know about the flatcar ignition fork vs the coreos ignition? I was looking at both today and as far as I can tell they are API compatible, but the coreos fork seems to be more active at the moment?

I think @pothos might know the answer.

[pothos]

The forked repo is only used by Flatcar LTS. Flatcar Stable uses the CoreOS upstream repo (with some downstream patches to also support Ignition spec v2).

[JoelSpeed]

Thanks @pothos, based on that, I think we should make sure we move back to the upstream coreos repo to import the types rather than using the forks, thanks for the context

[johananl]

We should consider this ☝️ in the context of #5294.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[johananl]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[johananl]

/remove-lifecycle stale

[FR-Solution]

waite this feature)

[fabriziopandini]

FYI
We currently don't have any maintainer for the ingnition code in the project 😓
Also, when we introduced ignition there was a general commitment to pay down the tech debt introduced by this choice before making further improvements in this area, but also for this effort there are no volunteers showing up (and it is a few years now)

Not really sure about what will be the next steps here (even if we keep the issue around)

[JoelSpeed]

Just a note, I think since I last commented here we added ignition 3.x support to CAPA. We are not users of the KubeADM provider but may be able to help with implementation guidance if one of the existing users wants to work on it.

[bengentil]

but also for this effort there are no volunteers showing up (and it is a few years now)

I submitted a PR addressing this feature but was advised to stop working on it as it would add additional dependencies and a larger rewrite was ongoing in this issue: #5294

#9158 (comment)

[Karthik-K-N]

Hi, Was there any further discussion about taking this forward in slack or CAPI meeting? interested in getting this feature added to CAPI.

[defo89]

Hi, since the roadmap and timeline of #5294 is not clear, would the community consider revisiting merging #9158?

Being stuck with Ignition v2 for another year or two means:

• not being able to use any modern OS with CAPBK unless an OS provides in-place v2 to v3 conversion (like Flatcar does)
• not possible to use kernel arguments as of 3.3.0 (and other v3 features, too)
• using legacy Ignition v2 that is deprecated long time ago

FWIW CAPBK should switch to Ignition v3 by default and base any other redesign work on it.

[fabriziopandini]

FWIW CAPBK should switch to Ignition v3 by default and base any other redesign work on it.

In the past a similar decision in this area did not lead to the expected results (see #9157 (comment))

Considering this, and the current bandwidth of the folks actively maintaining the project, I'm personally -1 to continue down the current path just because there are not enough people investing on this feature and in paying down the technical debt related to it.