[RFC #0102] Image history metadata

[nebhale]

Currently, when a layer is created from a Dockerfile, some additional metadata is added to the created image containing the directives that were run to create a particular layer:

Since buildpacks don't have "directives", this metadata is blank creating what I call a "black hole" for metadata:

The lifecycle should populate whatever metadata is necessary in order to fill in this black hole. As a straw man, I'd suggest each layer getting something like <buildpack name> <buildpack version> – <layer name> (e.g. Paketo BellSoft Liberica Buildpack 3.2.0 – jre) but I'm not really fussed with the actual content of this metadata.

[jromero]

Was looking a bit at this out of curiosity and found the the field used in the config is under history.created_by and it's described in the spec as "The command which created the layer.".

I wondered if there was any real considerations that would need to be taken into account like something trying to use this to recreate said image and there by needing to be executable. The nebulous description led me to explore certain usages and my ultimate finding was that there really isn't a definitive outcome. I'm only sharing this here in case it may help someone else.

Docker, using docker history cnbs/sample-stack-run:bionic --no-trunc, does current provide executable bits like:

/bin/sh -c mkdir -p /run/systemd && echo 'docker' > /run/systemd/container

as well as proper comments such as:

/bin/sh -c #(nop)  ARG cnb_gid=1000

but then this when using ARGS which I failed to find any real docs into it's format (non-executable?):

|2 cnb_gid=1000 cnb_uid=1000 /bin/sh -c groupadd cnb --gid ${cnb_gid} &&   useradd --uid ${cnb_uid} --gid ${cnb_gid} -m -s /bin/bash cnb

[ddobrin]

This is a question which is being asked by virtually every client of my company after demos.
Using the dive utility is an option, however is something totally different and can't be scripted.
Thank you, it would be very helpful to have

[jchesterpivotal]

1. A quick drive-by obligatory request for JSON formatted information for ease of slurping.
2. It can possibly be added as a shell comment to avoid execution.

Hence something like:

# {"buildpack_name":"Paketo BellSoft Liberica", "buildpack_version":"3.2.0","layer_name":"jre"}

Or even, alternatively:

echo '{"buildpack_name":"Paketo BellSoft Liberica", "buildpack_version":"3.2.0","layer_name":"jre"}'

[jromero]

The idea of making it parsable is interesting. I'm not particularly keen on it being JSON because it's not the most human-readable (as presented via docker history or dive). There are other formats that might be a good balance but have their own downfalls.

For example, comma-separated values:

# Paketo BellSoft Liberica,3.2.0,jre

or the distribution(?) spec can define a specific format that is both human readable and parsable based on a format contract.

format # <id>@<version> - <layer>

# Paketo BellSoft Liberica@3.2.0 - jre

[jabrown85]

Does anyone here want to create a draft RFC for discussion? I like the ideas presented here.

[jchesterpivotal]

CSVs are a famous disaster zone and I'm fairly militantly against inventing minilanguages. I see tool robustness as outweighing curiousity in this case. Especially since it's easier to submit PRs for dive et al to pretty-print JSON than to find every parser that inaccurately interprets an unspecified, informal minilanguage.

[samj1912]

Related - buildpacks/rfcs#194

[natalieparellano]

Blocking this on buildpacks/rfcs#194

Edit: actually, since the linked RFC is in FCP, let's just call it "ready"

[samj1912]

I can take this on :)

[natalieparellano]

Thanks @samj1912 !!!

[samj1912]

I have slowly been making progress on this (when I get time). I have started with some basic changes to imgutil (to allow setting and getting createdBy arrays through the image interface) and prepopulating this data from base image. Will update with a draft PR once it looks okay on the imgutil side.

[edmorley]

Will this issue cover adding metadata for builders generated by pack builder create too, or do we need a separate one for that?

eg: Currently our builder images show <missing> in docker history:

$ docker history test-builder-22
IMAGE          CREATED        CREATED BY   SIZE      COMMENT
6d4382092962   42 years ago                0B        
<missing>      42 years ago                45B       
<missing>      42 years ago                322B      
<missing>      42 years ago                6.53MB    
<missing>      42 years ago                6.14MB    
<missing>      42 years ago                15.3MB    
<missing>      42 years ago                11MB      
<missing>      42 years ago                13.4kB    
<missing>      42 years ago                32.5kB    
<missing>      42 years ago                5.6kB     
<missing>      42 years ago                10.7MB    
<missing>      42 years ago                4.46kB    
<missing>      42 years ago                25.6MB    
<missing>      42 years ago                15kB      
<missing>      42 years ago                6.01kB    
<missing>      42 years ago                13.4MB    
<missing>      42 years ago                0B        
<missing>      42 years ago                0B        
<missing>      42 years ago                335kB     
<missing>      42 years ago                375MB     
<missing>      42 years ago                1.73kB    
<missing>      42 years ago                543MB     
<missing>      42 years ago                9.14kB    
<missing>      42 years ago                77.8MB    

[natalieparellano]

@edmorley I think we'll want a separate issue for that on the pack side. This should be relatively easy to implement in pack once buildpacks/imgutil#202 lands.

[edmorley]

Will this issue cover adding metadata for builders generated by pack builder create too, or do we need a separate one for that?

I've filed:
buildpacks/pack#2052