CNB Roadmap 2021 Ideas

[hone]

It's that time for us to take a look at how the Cloud Native Buildpacks project is doing and see what we want to plan for the future. Whether you're using Buildpacks every day or not, we want to know your opinions. Your feedback here will help us understand where we are doing well, where are our gaps, and ultimately influence our priorities for the year.

For reference here is our 2020 roadmap with the accompanying blog post.

We'll be meeting as a leadership team late next week using this discussion. I look forward to see everyone's ideas!

[jromero]

More CI/CD integrations

Now that the specification is becoming stable, I think it would be beneficial if we could provide further integrations with common CI/CD platforms. As a user, having the ease of picking something off-the-shelf versus having to manually integrate it (even if the effort is minimal) can would make trying it out or incorporating it with existing infrastructure that much faster/easier.

Taken from an issue initially focused on pack:

Top 10 CI/CD platforms as ranked by stackshare:

• Jenkins
• Travis CI
• Circle CI (pack-orb)
• GitLab CI (AutoDevOp)
• Codeship
• TeamCity
• Bamboo
• Drone.io
• GitHub Actions (pack-cli-action)
• AWS CodeBuild

[schneems]

Agree. Heroku's v2 Buildpack API provides

• test-compile: https://github.com/heroku/heroku-buildpack-ruby/blob/main/bin/test-compile
• test: https://github.com/heroku/heroku-buildpack-ruby/blob/main/bin/test

It would be great to start working on a CNB compat version.

Edit: Strikethrough my comment because it's not the best place for it.

[hone]

This been something Gitlab has been interested in as well.

[schneems]

Ahh, I better understand the proposal. You want it to be easier for buildpack authors and maintainers to integrate with CI/CI systems? Not that you want buildpacks to be able to execute customer apps on ci/cd systems. I'll make that a separate proposal and remove my above comment.

[hone]

@schneems I think they're related.

[waldemar-kindler]

I think we can add Azure DevOps to that list as well. :)

[ekcasey]

Asset Caches

[ekcasey]

Develop API

[jromero]

@schneems I think this is more inline with your comment.

In my mind a Develop API could enable two use cases:

1. Providing a phase in which tests can occur and potentially their outputs be omitted
2. Providing a "quick feedback loop" for applying changes from source code to something that can be inspected. (think live reloading)

The latter would likely require more of a platform implementation to provide that level of support but buildpacks can enable it by provided the necessary information and/or hooks.

[schneems]

I also had a recent run-in with a customer who was asking for a CNB dev story. Internal ticket number 961206. From the customer

Our goal is to have an image as similar as possible with the image in production (ruby/node and libraries) but using it for development.

Which makes sense to me. It seems silly to have this awesome tool that can be used in prod and test, but not dev.

For Ruby it would need to support code reloading (RAILS_ENV=development instead of RAILS_ENV=production) as well as installing all dependencies (instead of BUNDLE_WITHOUT="test:development"). In an ideal case someone would be able to clone an app, run a command and get a working dev environment with no additional steps.

External service dependencies might be the biggest unknown (i.e. postgresql, redis, etc) along with durability.

[ekcasey]

In my mind a Develop API could enable two use cases:

Providing a phase in which tests can occur and potentially their outputs be omitted
Providing a "quick feedback loop" for applying changes from source code to something that can be inspected. (think live reloading)

I was thinking about the "quick feedback loop" use case in particular when I threw this on here. In my mind these two use cases are similar in that they are both important to the development workflow, but they are also distinct enough from each other that I think we should be talking about a "Develop API" and a "Test API" separately to avoid confusion.

[ryanmoran]

The Paketo Buildpacks team has been looking into how we can integrate buildpacks more fully into existing developer workflows. To start, we did some exploration of Tilt and Skaffold to understand their existing support for buildpacks, and how we might build a better experience if we enabled these types of workflows more directly in the spec.

Here are some quick writeups of our initial explorations: Tilt and Skaffold. We'd like to see some of the ideas discovered here in a future "develop API".

[schneems]

Agreed on separate test and develop. Rails comes with three environments baked in: development, test, and production. The default for buildpacks is production. I find having a separation between development and test is really nice and allows for optimizations.

[ekcasey]

Spec Refactor

On the roadmap with can be appropriate vague here but in preparation for 1.0 we should clean up the following aspects of the spec to improve predictability/comprehensibility make future additive changes easy:

• Extract Stack API spec from platform spec
• Refactor platform/buildpack/lifecycle(?) specs so the requirement for each are separate and clear
• Consistent and well defined terminology
• Consistent schemas for all *.toml files
• Consistent image label Schemas
• Organization of input/output files to lifecycles including different locations for
  • Inputs provided by the platform (either directly on or behalf of a user) to buildpacks
  • Inputs provided by the platform (either directly on or behalf of a user) to the lifecycle
  • Default inputs provided by the builder
  • Inputs provided by a previous lifecycle phase

[Malax]

This is spot-on, especially with the current state of the documentation I find myself referring to the spec a lot and you summarized the issues pretty well I think. :)

[sampeinado]

I agree strongly with @Malax 's point. Today, to get things done when authoring a buildpack, whether the intent is for the buildpack to be reusable by others or just a one off to make your non-standard build work, users need to refer to the spec because documentation is just not there. We should consider making it a goal of this spec refactor effort to take a “docs driven development” approach to it, to make sure the spec’s powerful capabilities are clear enough to be taken advantage of by users. @nebhale had an interesting comment on this from one of the early user research findings docs which I’d like to repeat here:

“I think we need tiered user docs. We talk about the progressive complexity of the spec, but at every entrypoint into CNBs we give users the full complexity of the specification. We should actively be hiding certain parts of the spec from the docs, and putting it behind a "learn more" gate in order to make it more approachable.”

[ekcasey]

Builder specification - builders as a first-class citizen

[hone]

Does this mean not as an extension or just finishing up the spec work?

[ekcasey]

I guess I really meant second-class citizen (extension spec). Right now it is an undocumented third-class citizen :p

[ekcasey]

Better Documentation

[ekcasey]

I have specific thoughts on the existing pain points:

1. Current organization and nesting of pages is confusing. I have trouble predicting whether the information I am looking for is in tools or concepts or a guide. Examples (non-comprehensive):
   • Some of the top level content feels very pack specific so I have trouble understanding why some content goes in tools/pack vs. the App Developer's Guide
   • Why is guidance on how to crate a stack in concepts/stacks instead of a Stack Authors Guide?
2. Missing lifecycle docs
3. Missing guide for platform authors
4. We have a tutorial for buildpack authors, but I think we need more reference-style material.
5. We add docs as we add features (much better than what we used to do), but could afford to back up and fill in older missing information. For example we have a section on cache images but never describe the (default) volume cache workflow (unless it in there and I just haven't found it yet).

[schneems]

I would like to see more tasks specific tutorials similar to https://guides.rubyonrails.org/. I think https://buildpacks.io/docs/buildpack-author-guide/ is a good start, but there could be more.

[jromero]

Created a discussion just for this topic: #67

[sampeinado]

One piece of feedback that came out in our user research: our documentation today is mainly oriented toward understanding the spec, which makes things harder for users trying to quickly take advantage of the spec to troubleshoot their failing (or improperly building) build. A couple suggestions from a field engineer at VMware:

• Make it a goal for every docs page to have actionable, copy-paste-able things in addition to explanation
• Organize the spec documentation around tasks / troubleshooting steps you can accomplish using knowledge of the spec
• Expand the buildpack author documentation, the one tutorial we have today “leaves you alone as soon as you want to go off the rails”

[sampeinado]

More documentation feedback from our user research process: @natalieparellano and I got feedback from buildpack novices on the process of customizing buildpack builds by asking the participants to replicate this Dockerfile build with CNBs.

• One of the challenges Natalie ran into when demoing the exercise was the lack of guidance on how to add things to the path. She wrote:

“I struggled to add dockerize to the PATH (because I was putting files in /env instead of //env) and it got me looking for documentation on how to do this - buildpacks.io mentions setting env vars, but not prepending or appending to PATH with delimiters, and the spec does mention it, but very obscurely. Also reading the spec, I almost expected EVERY /layers/buildpack/layer-dir to be on the PATH.”
(Natalie is on vacation this week, so I am taking the liberty of sharing for her)

• Another challenge was in the absence of documentation about the launcher. She wrote:

“I added a “list-env” process type to use for debugging. I could have run the image with the launcher as the entrypoint and env as the command, but I was suspicious of my knowledge of how the launcher works. We might provide this as a debugging tip in our docs. It would be nice if you could set an environment variable in the image to have the launcher print more information like the exact command it’s running.”

[natalieparellano]

Support ARM

[schneems]

Cleanup step after compile

Backstory: The heroku/ruby buildpack runs some webpack commands (via rails integration), which rely on installation of node modules, however the heroku/node buildpack cleans "dev dependencies" after it's done executing. This means that for Ruby to execute node first installs modules, then uninstalls dev dependencies, then Ruby has to re-install them, then it can run webpack.

To fix this we could split the node clean logic into yet another buildpack, but it gets tiresome doing this and puts the burden on the "upstream" buildpack (from the perspective of heroku/ruby). I've informally tossed around the idea of a "cleanup" phase that would run after all buildpacks run. For example Node could register that it wants to clean up dev dependencies, ruby can register that it wants to execute bundle clean and other buildpacks can register their own commands. After all buildpacks execute successfully then the clean phase would run.

This would benefit this use case because dev dependencies would not be deleted yet. As a side benefit builds that fail would get to fail faster since the cleanup steps wouldn't have to run.

I'm thinking we could have buildpacks write cleanup command(s) to their TOML and then have lifecycle execute them after all buildpacks succeed.

For bonus points, we could maybe have buildpacks declare cleanup commands as being able to run concurrently or not, to speed stuff up, but that gets dicey if two buildpacks think they can both modify/delete the same entry in node_modules then it will trigger OS errors.

[thephw]

I think driving adoption is still top of mind for me. I don't have any quick solutions, but anything that helps drive adoption is going to help make a better business case for buildpack builds vs Docker builds. Docker has a superior market position and it's still a big challenge to unseat the incumbent. I feel that amplifying social proof and content marketing is still really important -- demonstrating that adopting buildpacks provides better developer productivity, operational security, and a wider hiring market for developers (less ops knowledge needed to be productive). Measuring the audience on our outreach could be a good first step into focusing those efforts.

[dmikusa]

I wish I could upvote this more. Sharing a vision about how someone can use this great technology is important. Not just how folks can participate in the project, or write their own buildpacks. We have to spend more time showing how problems can be solved with buildpacks, how buildpacks can be integrated into existing systems so there's a more complete picture, and driving home the benefits.

[sampeinado]

Based on our user research process and findings, I see a number of opportunities here. We did a feature ranking exercise with each of our interviewees, asking them to rank the features of a container build system in order of importance. When averaged across the 9 participants, the top two most important features were both related patching/updates:

1. Ability to patch security vulnerabilities in application dependencies
2. Ability to patch security vulnerabilities in OS packages and language runtimes

Based on this finding, i think we really need to emphasize how much better the experience of doing an update to a CNB image is than an update to a dockerfile. Here is a quote from one of the participants, a Devops engineer that describes how "time to update" affects "time to market:"

“Security patches and all, it is a little complicated and time consuming and sometimes it is not happening end to end and the team struggles. But if this is becoming flexible, it will be very helpful and speed up delivery of our product. Sometimes we are not being able to test the build on time (devs sign off on the build, testers signing off on the build) and that will prevent you delivering the product in a certain span of time. You have to follow up with the reviewer and ask them to please finish it.”

In theory, an update done via CNBs would avoid this manual approval process that has to take place for every little change to the dockerfile

[sampeinado]

The 3rd most important feature from this ranking exercise, when averaged across the 9 participants, was speed of builds. Based on this finding, I would argue that we need to do everything in our power to ensure the first build is FAST (10s? Certainly <60s). @nebhale suggested an idea on how the lifecycle could help with this in one of the early user research findings docs which I’d like to repeat here:

“the problem is compounded if that build fails after a slow bit because caches are all or nothing rather than incremental. We should prioritize the change to spec/lifecycle around cache validity.”

[sampeinado]

The 4th most important feature from this ranking exercise, when averaged across the 9 participants, was "total control and flexibility for customization". Based on this finding, I would argue that we really need to invest in flexibility and "escape hatches," which I will discuss more in a new thread.

[sampeinado]

Finally, a couple surprising findings: we need to make clear that CNB images can be used on kubernetes. I know, I know, it seems obvious, but based on our research, developers and devops engineers are not very familiar with the term “OCI image” and not all are able to make the connection that CNB containers can run just as well as docker on kubernetes (one developer straight up asked: “is it compatible with k8s, do you need k8s to run it?”).

In addition, we need to make clear that while CNBs scan your app code to detect dependencies, CNBs do not store information about your app, nor do they send it back to any database. Two of our participants expressed concern about the security implications of the ‘detect’ phase. One explained the security risk as:

“If someone is analyzing our code, on some tiny level they could inject something or copy the code”

while another worried about key privacy:

“Does it show that the tool is not sending to the cloud or somewhere? We have all our keys locally, so the tool would then be analyzing all of our local keys if its building from there.”

[wburningham]

Testpack spec. This is something that heroku does and I would love to see an open-source version spec'd out.

[schneems]

Yes! I accidentally said this in a different proposal

test-compile: https://github.com/heroku/heroku-buildpack-ruby/blob/main/bin/test-compile
test: https://github.com/heroku/heroku-buildpack-ruby/blob/main/bin/test

It would be great to start working on a CNB compat version.

[mhoran]

I'd love to see this. Buildpacks are a great alternative to Dockerfiles, but unfortunately the current situation requires that you still use Dockerfiles or similar to set up your CI environment (unless using Heroku's CI offering). I see a number of benefits from being able to build a test layer or something similar, which is stripped out for images run in production.

1. The test layer will provide test dependencies on top of the same layers that are used in the production image. This would prevent any sort of runtime (e.g. MRI) discrepancies from impacting tests in any way.
2. Having a test layer in the output of a CNB image would mean that automated tests coudl be run against rebased images (think: update MRI). This goes hand in hand with 1) in that, if an MRI change breaks a system, this could be known before an image is deployed to production.
3. Testpacks would reduce duplication. The Dockerfile needed for CI in this case is ultimately declaring the same thing that you get through the Buildpack process. It is in the interest of DRY to have the Buildpacks process output an image that can be used for both test and deployment.

[sampeinado]

Pack compose

100% (6 out of 6) of the developers and Devops engineers we interviewed in the user research process reported that they use docker compose as much as docker build. One of the participants described his compose.yml as "really rich" saying that he includes:

“Env variables, ports, and services, like If service depends on database we have to mention it.”

@ekcasey mentioned during the initial user research scoping session that this was a feature we are considering adding to pack and based on this finding, I think it would be worthwhile to invest to add it.

[dfreilich]

Totally so. I think that this could, hopefully, be accomplished with the Develop API mentioned above – #62 (comment)

[sampeinado]

Flexibility and "Escape Hatches"

Today, buildpacks are designed to be reusable, and that's a huge chunk of their value. That said, it's important to note that dockerfiles are also reusable (people copy and paste them all the time). The dockerfile "escape hatch" when you reuse something is to simply delete and rewrite a few (or more) lines to fit your specific app's needs. I think that in order to serve more developer use cases, we need to invest in our "escape hatches" for using CNBs.

I see three levels at which we have an opportunity to have better flexibility for buildpack builds:

• Better explanation of the purpose and practical use of environment variables and service bindings (docs, tutorials, and example walkthroughs)
• A solution for creating one-off custom buildpacks quickly to unblock yourself (potentially inline buildpacks)
• An easier path to writing your own reusable buildpack when you believe the investment is justified (and easier access to the existing work of others - ex. the Buildpacks Registry)

[sampeinado]

Here are some more quotes about why flexibility is so important to developers and DevOps engineers:
When I asked developers about their comfort using a build tool that had the logic already written for them, several expressed concern about:

“how much of an escape hatch there is, i would want to make sure that the system would allow me to do something like a very unconventional networking implementation”

And another was fearful about a:

“Lack of flexibility if you’re trying to build something unique”

Based on his previous experiences with other automation tools, he knows that:

“Only using the tools’ language means there will be some nuances, like the terraform language, there’s no way to get around those nuances. A declarative language is very easy to use, but if you have a problem you’re going to really need to scratch your head to think through how to solve it”

It would be easy for us to respond to this and say that, “buildpacks and the lifecycle aren’t a black box! They’re mostly open source so you can just go read the code!” But let’s remember that they are written in Go and most developers don’t know or feel comfortable reading Go (On 1 to 5 scale of their comfort with writing Go, Docker users we interviewed averaged a 2 out of 5 ). It’s a very “Silicon Valley” programming language, which is really only a small subset of the developers and devops engineers we want to serve.

Furthermore, expecting our users to have the time or interest to read our undocumented code is unrealistic. That may be ok for the group who works on this project since y'all are a bunch of really brilliant engineers who not only use cutting edge tools but build them and define what the cutting edge is.

For us to be successful as a project, we need to make buildpacks a tool that is understandable for average and below average developers - the sort of people who don’t know what yaml is, who are overworked bc their colleague just quit and they have to do the work of 2 people, and even the people who are just in it for the paycheck and arent constantly striving to do their job better. To get to the next level of adoption growth, we need to meet our users where they are.

[sampeinado]

With respect to environment variables, we need to add a number of basic use cases to our documentation. Some of the challenges @natalieparellano ran into when she practiced the Dockerfile replication test were:

• The “environment variables” section in docs don’t make it clear that it’s build time only; could change the header to “Build time environment variables”
• Docs don’t describe how to set a runtime environment variable, at all
• It’s not clear that /bin/build is part of buildpack code vs app code, given that a usability test interviewee asked “will we need to hard code these in the app”?

In addition, the very concept of setting an env var or binding a service as a way to customize your container build is not a part of our documentation. For example, "add dependencies by setting env vars" is not a step in our getting started docs. This capability is only usable if a novice buildpacks user understands that specifying env vars results in new dependencies being added to your image.

[sampeinado]

With respect to “one-off custom buildpacks,” one of the annoyances that @natalieparellano ran into was in clearing the cache for a specific layer while developing a buildpack. She wrote:

“I tried to cache dockerize but when I made changes to the dockerize layer, I forgot to clear the cache and got very confused as to why my buildpack wasn’t working. With Dockerfile (because changing the command busts the cache) this isn’t a concern. Separately, it would be nice to clear the cache just for a specific layer, because I had to re-download Java to fix my dockerize layer.”

In general, she found that:

“My experience writing this buildpack was of making one non-obvious typo after another.”

In order for us to develop empathy with this challenge, I would propose we should have a weekly or monthly session where we observe a non-contributor developer try to build their app with CNBs and see where they struggle. This would give us a lot of examples of how we can improve the “first time build” experience. But we can't improve that experience if we keep treating ourselves (buildpack experts) as if our experience is the same as it is for buildpacks novices. We need to bring buildpacks novices into our process to keep us honest.

[aemengo]

Buildpackfile

I’d argue that a significant amount of people that use Dockerfiles, or Vagrantfiles that preceded it, don’t fully understand them. But they’re easy to copy and paste. Easy to change one line.

I’m not saying it even has to be a file, just that intuitive experience.

[jkutner]

I think Buildpackfile might similar or related to Inline Buildpacks, which haven't been implemented yet

[sampeinado]

Tool to Visualize the Build Process

Several participants in our user research expressed desire to really understand how the container is built by buildpacks, both in order to have more confidence in the system and also to make troubleshooting easier:

“It’s like a personal satisfaction. I need to know why I am doing each thing so that I know what’s happening, that this does this and that does that and I’m not blindsided by something that goes wrong later, I know the purpose.”

To that end, two of our participants separately proposed a system for visualizing the buildpacks image construction process. One, a DevOps consultant said:

“The build system should provide the events and then the four stages could be visualized - detecting, choosing the buildpack, etc - then resource logs, how many objects, which objects have been touched, cached, rebuilt over time, why they have been rebuilt. And you can connect to this tool at any time, so you can attach to a build that is going slow and see what’s going on. You can inspect the variables that lead to this point, that would be useful for troubleshooting, being able to get stuff out of the containers to understand what goes on there.”

This DevOps consultant also suggested that a visual system like this could function as an information radiator, similar to CI pipelines displayed on big tv monitors, and that would help to make the DevOps process more collaborative. He encouraged us to:

“Make this build system a toy for everyone like in the past people watch the deployment screens”

Another participant suggested that such a tool that deepens understanding of how buildpacks actually work would help build confidence in our security posture, since his

“Biggest concern [about buildpacks] is security.”

[abitrolly]

Also here https://github.com/buildpacks/rfcs/blob/main/text/0092-add-visual-pack-build.md

[sampeinado]

Hey folks! Just a heads up @natalieparellano and I are going to lead a 2-part Research Shareout Workshop during CNB Office Hours on Thursday 5/6 and Thursday 5/13. Hope to see all of you there!

[natalieparellano]

Superseded by #200