More fixes for target compat checking during detect #1354

natalieparellano · 2024-05-13T19:18:07Z

Summary

If a buildpack does not specify targets, but DOES specify stack with ID io.buildpacks.stacks.bionic, only translate that to a target if the buildpack is on a newer Buildpack API (where stacks are deprecated, and we expect targets) OR if the buildpack only ever declared one stack. When multiple stacks are declared, specifying only one target actually reduces the number of builds that can succeed. Older Buildpack APIs don't actually specify stack -> target translation so we should do the thing that will work in the maximum number of builds.
If a buildpack fails to specify os/arch (but specifies distro) still check targets
If the run image fails to specify os/arch (this should not happen actually as we will fail during analyze) still check targets
Fix typo in buildpack descriptor struct ~~so that we actually get stack information~~ (we still see it because of the struct field name)
If we get distro information from /etc/os-release, persist this information to later invocations so that the log message printed when errors are encountered will be accurate
Don't override inner i in loop (this should not actually affect the outer loop but is confusing)

Release notes

If a buildpack does not specify targets, but DOES specify stack with ID io.buildpacks.stacks.bionic, the lifecycle will only translate that to a target if the buildpack is on a newer Buildpack API (where stacks are deprecated, and we expect targets) OR if the buildpack only ever declared one stack
The detector, when checking target compatibility, will still perform the check if buildpacks fail to declare os/arch information in targets

- If a buildpack fails to specify os/arch (but specifies distro) still check targets - If the run image fails to specify os/arch (this should not happen actually as we will fail during analyze) still check targets - Fix typo in buildpack descriptor struct so that we actually get stack information - If we get distro information from /etc/os-release, persist this information to later invocations to that the log message printed when errors are encountered will be accurate - Don't override inner `i` in loop (this should not actually affect the outer loop but is confusing) Signed-off-by: Natalie Arellano <narellano@vmware.com>

Missing targets is sufficient for wildcard match Signed-off-by: Natalie Arellano <narellano@vmware.com>

robdimsdale · 2024-05-13T20:08:54Z

buildpack/bp_descriptor.go

@@ -70,8 +70,6 @@ func ReadBpDescriptor(path string) (*BpDescriptor, error) {
 		for _, stack := range descriptor.Stacks {
 			if stack.ID == "io.buildpacks.stacks.bionic" {


should we add jammy given it's the primary stack that paketo supports?

As discussed - we will opt for a len == 1 && stack == bionic check, as bionic is encoded in the spec.

robdimsdale · 2024-05-13T20:10:57Z

phase/detector.go

-			targetMatch := false
-			if isWildcard(d.AnalyzeMD.RunImageTarget()) || hasWildcard(descriptor.TargetsList()) {
+			var targetMatch bool
+			if len(descriptor.TargetsList()) == 0 {


Is this for backwards-compatibility with buildpacks that don't have a targets field? Is this spec-compliant?

In reality, this is for tests. In practice, the lifecycle will infer at least an OS when targets are missing. Here is what the spec has to say about matching:

If the targets list is empty, tools reading buildpack.toml will assume:

os = "linux" and arch = if ./bin/build is present
os = "windows" and arch = if ./bin/build.bat or ./bin/build.exe are present
Metadata specified in [[targets]] is validated against the runtime and build-time base images.

A buildpack target satisfies a base image target when os, arch, and variant match and at least one distribution in distros (if provided) matches

… succeed Fixes #1355 Signed-off-by: Natalie Arellano <narellano@vmware.com>

Signed-off-by: Natalie Arellano <narellano@vmware.com>

natalieparellano · 2024-05-14T16:36:27Z

acceptance/testdata/detector/Dockerfile

@@ -1,4 +1,4 @@
-FROM ubuntu:bionic
+FROM ubuntu:jammy


This change causes

lifecycle/acceptance/detector_test.go

Line 163 in 1a1de08

it("writes group.toml and plan.toml at the default locations", func() {

to start failing when

[[stacks]] id = "io.buildpacks.stacks.bionic" [[stacks]] id = "io.buildpacks.stacks.jammy"

is added to acceptance/testdata/detector/container/cnb/buildpacks/simple_buildpack/simple_buildpack_version/buildpack.toml without corresponding changes in how we read the descriptor

natalieparellano · 2024-05-14T16:37:21Z

buildpack/bp_descriptor.go

@@ -17,7 +18,7 @@ type BpDescriptor struct {
 	Order       Order            `toml:"order"`
 	WithRootDir string           `toml:"-"`
 	Targets     []TargetMetadata `toml:"targets"`
-	Stacks      []StackMetadata  `tome:"stacks"` // just for backwards compat so we can check if it's the bionic stack, which we translate to a target
+	Stacks      []StackMetadata  `toml:"stacks"` // just for backwards compat so we can check if it's the bionic stack, which we translate to a target


Worth noting that this typo while unsightly didn't cause errors because the struct field name matches the expected TOML key name

natalieparellano · 2024-05-14T16:38:56Z

phase/detector.go

@@ -144,26 +144,14 @@ func (d *Detector) detectOrder(order buildpack.Order, done, next []buildpack.Gro
 	return nil, nil, ErrFailedDetection
 }

-// isWildcard returns true IFF the Arch and OS are unspecified, meaning that the target arch/os are "any"
-func isWildcard(t files.TargetMetadata) bool {


In practice, isWildcard would never return true because we check that we at least have an OS during analyze (also OS/arch is required in the OCI spec)

natalieparellano · 2024-05-14T16:39:43Z

phase/detector.go

-	return t.Arch == "" && t.OS == ""
-}
-
-func hasWildcard(ts []buildpack.TargetMetadata) bool {


It's possible to have a target without OS/arch but with a distribution so this function never made much sense

natalieparellano · 2024-05-14T16:40:05Z

phase/detector.go

 func (d *Detector) detectGroup(group buildpack.Group, done []buildpack.GroupElement, wg *sync.WaitGroup) ([]buildpack.GroupElement, []files.BuildPlanEntry, error) {
 	// used below to mark each item as "done" by appending it to the done list
 	markDone := func(groupEl buildpack.GroupElement, descriptor buildpack.Descriptor) {
 		done = append(done, groupEl.WithAPI(descriptor.API()).WithHomepage(descriptor.Homepage()))
 	}

+	runImageTargetInfo := d.AnalyzeMD.RunImageTarget()


This is so we can keep track of the info we read from /etc/os-release and print a more meaningful log message

natalieparellano requested a review from a team as a code owner May 13, 2024 19:18

natalieparellano marked this pull request as draft May 13, 2024 19:22

When stack is "any", don't infer empty target as it is not needed

4f98bcf

Missing targets is sufficient for wildcard match Signed-off-by: Natalie Arellano <narellano@vmware.com>

robdimsdale approved these changes May 13, 2024

View reviewed changes

natalieparellano added 2 commits May 14, 2024 12:33

Remove backwards compatible glue that actually causes fewer builds to…

5b20fbe

… succeed Fixes #1355 Signed-off-by: Natalie Arellano <narellano@vmware.com>

Remove exit (this was added for debugging purposes)

e3d66c8

Signed-off-by: Natalie Arellano <narellano@vmware.com>

natalieparellano commented May 14, 2024

View reviewed changes

robdimsdale approved these changes May 14, 2024

View reviewed changes

natalieparellano marked this pull request as ready for review May 14, 2024 17:37

jabrown85 approved these changes May 14, 2024

View reviewed changes

natalieparellano merged commit 73f6927 into main May 14, 2024
8 checks passed

natalieparellano deleted the fix/target-data branch May 14, 2024 18:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More fixes for target compat checking during detect #1354

More fixes for target compat checking during detect #1354

natalieparellano commented May 13, 2024 •

edited

Loading

robdimsdale May 13, 2024

robdimsdale May 14, 2024

robdimsdale May 13, 2024

natalieparellano May 13, 2024

natalieparellano May 14, 2024

natalieparellano May 14, 2024

natalieparellano May 14, 2024 •

edited

Loading

natalieparellano May 14, 2024

natalieparellano May 14, 2024

		@@ -70,8 +70,6 @@ func ReadBpDescriptor(path string) (*BpDescriptor, error) {
		for _, stack := range descriptor.Stacks {
		if stack.ID == "io.buildpacks.stacks.bionic" {

More fixes for target compat checking during detect #1354

More fixes for target compat checking during detect #1354

Conversation

natalieparellano commented May 13, 2024 • edited Loading

Summary

Release notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natalieparellano May 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natalieparellano commented May 13, 2024 •

edited

Loading

natalieparellano May 14, 2024 •

edited

Loading