Feature request: Build args on docker hub

[cusspvz]

Since we can have ARGS on dockerfile, seems right to have them available on auto-builds as also (for having different tags based on the same Dockerfile)

[cheungpat]

Together with LABEL, build args is very useful to apply custom metadata to images. For example:

FROM debian:wheezy

ARG branch
LABEL com.example.branch=$branch

Setting --build-arg branch={sourceref} can add the com.example.branch=master label to the image. It would be great if Docker Hub supports more build variables to provide metadata such as build id, git commit hash, creation time etc.

[azenla]

Yes, it would be really useful. +404003

[andyneff]

I've been thinking the same thing since I started using buildargs. It would be perfect if there was a way to set specific buildargs for specific tags on my dockerhub image, instead of having to create separate tags/branches in my git repo.

[yajo]

This would be even better than #292

[cusspvz]

@yajo note that this is regarding access to set --build-args option and not entire docker build cli args.

[yajo]

Yes, I just meant that #292 could cover the same use cases, but fixing this would be much better for most of those.

[schickling]

This would make multi-version maintenance of images a lot easier

[pchico83]

This is available in Docker Cloud, which uses the same backend than Hub for autobuilds. The way to do it is to define autobuild env variables, an write a build hook that does the docker build command with the required build arguments.
https://docs.docker.com/docker-cloud/feature-reference/automated-build/
https://docs.docker.com/docker-cloud/feature-reference/automated-testing/

[nalipaz]

Can someone explain the method described by @pchico83? Because the documentation does not mention anything about writing a build hook, only hook/pre_build and hook/post_build as part of testing from what I can tell.

[pchico83]

@nalipaz although it is not documented, it is supported (I will open an issue to improve the documentation). For example, if you define an env var called VAR and you want to set it to a build argument called TEST, you can define a build hook adding a hooks/build file to the dockerfile folder:

.
..
hooks/build
Dockerfile
(other files)

whose content could be:

#!/bin/sh
docker build --build-arg TEST=$VAR -t $IMAGE_NAME .

Note that the hook is executed using the dockerfile folder as working dir, and $IMAGE_NAME is a variable we inject that corresponds with the docker tag being pushed.

[nalipaz]

That is much clearer, thanks @pchico83, and yes it would be very helpful to many I am sure were it documented. Thanks.

[Gurubol]

@pchico83, @nalipaz , Can you please provide me an example of how this hooks is used? Do we use the same docker file commands?
I tried with an example of setting an environment variable, which I later want to refer to in the Dockerfile, but it throws error

For example, in hooks/post_checkout, I added

ENV UI_BUILD x-1.2.2

But this throws me the following log.
"Executing post_checkout hook...
Unexpected error"
Can you please throw some lights on this? Any help is appreciated.

[nalipaz]

Yeah, a gist or example project would likely be helpful for this. I will see if I can find time to test and make one although I am not using this feature currently. I opted for using Travis CI to use a make file to pass in params to the build and then push the final image into docker hub.

[pchico83]

@Gurubol You are probably missing the bash shebang at the beginning of your hook.
This is an example project with hooks:
https://github.com/docker/dockercloud-events

[andyneff]

Thanks to @pchico83 guidance, I was able to look around and discovered a number of useful pieces of information

1. The docker image used in docker hub to build dockers is called docker/highland_builder
2. The majority of the hidden hooks features are come from the builder.py
3. Here is a simple hooks example
4. There are a number of key environment arguments set upon launching a build script, all of which you can use in your hooks.

BUILD_PATH=/
MAX_LOG_SIZE=67108864
SOURCE_BRANCH=master
DOCKER_REPO=index.docker.io/andyneff/docker_hook_test
DOCKERCFG=REDACTED
LOGS_POST_SPEC=REDACTED
BUILD_CODE=b9manhaqhfyqtmdqupcjyuq
README_POST_SPEC=REDACTED
SOURCE_TYPE=git
DOCKER_HOST=unix:///var/run/docker.sock
DOCKERFILE_POST_SPEC=REDACTED
DOCKER_TAG=latest
PUSH=true
PYTHONUNBUFFERED=1
SOURCE_URL=https://github.com/andyneff/docker_hook_test.git
DOCKERFILE_PATH=
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DOCKER_VERSION=1.11.2
COMPOSE_VERSION=1.7.1

Redacted fields contained what looked like keys I assume would be harmless and expired by now, but I'm not sure ;)

But the take away is also have access to SOURCE_BRANCH and DOCKER_TAG, in addition to IMAGE_NAME, GIT_SHA1, GIT_MSG, and COMMIT_MSG, which can all be useful in making custom build-args :)

Hope this helps with this pretty awesome undocumented feature!

[t-botz]

Here is a test image I use to check which hook are triggered when and which variables are available :
https://github.com/thibaultdelor/testAutobuildHooks

Check the result of the build here :
https://hub.docker.com/r/thibaultdelor/testautobuildhooks/builds/

[andyleejordan]

I really need this. I have multiple Dockerfiles because I have multiple distros to support, so they're each in their own folder, meaning they don't have the whole repo in their context, so the repo is cloned during docker build and I just need to be able to pass ⁠⁠⁠⁠--build-arg branch=$(git rev-parse HEAD) in the automated build so that I can checkout the right commit. Argh.

[andyneff]

@andschwa I prefer to have multiple named Dockerfiles in the root of the repo and then just run docker build -f myrepo.Dockerfile and not need to copy the repo. Either way, you can achieve this using my post above. See here for a full example when we are doing almost exactly what you are referring to. Just customize the hooks/build to your hearts content. Hope this helps

[huan]

need this feature for the automatically build, would be really nice.

[edify42]

@nalipaz although it is not documented, it is supported (I will open an issue to improve the documentation). For example, if you define an env var called VAR and you want to set it to a build argument called TEST, you can define a build hook adding a hooks/build file to the dockerfile folder:

.
..
hooks/build
Dockerfile
(other files)

whose content could be:

#!/bin/sh
docker build --build-arg TEST=$VAR -t $IMAGE_NAME .

Note that the hook is executed using the dockerfile folder as working dir, and $IMAGE_NAME is a variable we inject that corresponds with the docker tag being pushed.

I think this is the fix and the issue should be closed?

[khalilgharbaoui]

@edify42 Well it could be closed, but the workaround does not actually supply the requested feature.

Right now it is possible to add BUILD ENVIRONMENT VARIABLES. But, in reality, those aren't really only for build... they are just usual ENV variables.

This naming is very confusing IMO and it should be renamed to ENVIRONMENT VARIABLES instead.

And, in addition, we need a BUILD ARGUMENTS section where those can be filled in.

Because there is a clear difference between the two: https://vsupalov.com/docker-arg-vs-env/

TL;DR
The workaround with hooks is great and works fine, but it adds overhead. And the naming on dockerhub (cloud.docker.com) is just wrong.

[bitsofinfo]

2015... and this still isn't available?

[abdennour]

since 2015 till 2019/09/26 .. still not available ! :(

[mitar]

They are available, you just have to use build hooks.

[mitar]

https://docs.docker.com/docker-hub/builds/advanced/

[edify42]

https://docs.docker.com/docker-hub/builds/advanced/

See my example...

[kingdonb]

Does this need to still be open? I see some thumbs down, but https://docs.docker.com/docker-hub/builds/advanced/#override-the-build-phase-to-set-variables actually solves this on Docker Hub. I was able to build an image with RAILS_MASTER_KEY set at build time, so I could run

RAILS_ENV=production bundle exec rake assets:precompile

and it worked absolutely fine! No sign of my build arg in the image output, which is the point of build-arg as I understand it. So, how are these ENV variables and not ARG variables? There is no sign of them in my build context either, unless I invoke the ARG keyword (in which case they are silently made available and not included in the image output, from what I can tell.)

To put a finer point on it, https://stackoverflow.com/a/55839775/661659

If you follow this example exactly, but in "step3" except instead of using ARG followed by ENV you simply call ARG with the name of the build-arg that you added into the hooks/build file, (per the same example previously linked by @edify42), you get a build-time only variable, it seems, unless there is something that I missed.

@khalilgharbaoui can you clue me in if there's something I didn't understand here, or can the maintainers feel free to close this? I think it's solved.

agh, maybe not quite secret, I guess this is what you meant:

<missing>           7 minutes ago       |1 RAILS_MASTER_KEY=c06afc06a31b6d17bf62361f…   180MB
<missing>           13 minutes ago      /bin/bash -l -c #(nop)  ARG RAILS_MASTER_KEY    0B

Is there a way to do this as we intended in Docker at all, or is it a request against nature?

I personally really did think this was an impossible request until I found numerous posts describing ARG as a way to put secrets into the build without letting them leak out through the images. But here it is, (at least most of) the secret (edit: yes, you can recover the whole secret and any other ARGs using docker history --no-trunc, here I see them in my docker history for the image, which I just built on Docker hub and pulled from there onto my docker machine.)

It looks like they are leaked out through docker history, but not into the container image runtime context. "So I've got that going for me at least, which is nice."

TBH this might still be marked as solved, I'm not sure how Docker images can maintain a consistent format if they allow secrets to be used at build time but not included in docker history output reports. This strategy prevents them from being available inside of the image at runtime, which if you want those secrets to remain private you probably TBH shouldn't publish images that are based on them to public registries anyway.

If you really want to use a secret and then don't include it in the build output, you could use ARG together with a multi-stage build and then squash, but actually I think maybe squash image is still an experimental feature. Thoughts?

[JuliusSustarevas]

Just wanna check what the current situation is:

• You can have a hooks/build where you "overwrite" the build command and can set your worn --build-arg.

But this method also overwrites the whole build command. Which means it also overwrites the --tag,--file, and most importantly --no-cache

• So if I pass build-arg by having a custom hooks/build. The button "use cashe" on "configure autobuilds" on docker hub, will do nothing?

[HarelM]

I have read every message in this thread and it's still not clear to me if this is a problem in my code or a bug in dockerhub.
My use case is as follows (a very straight one from my point of view):
I set up two automation build in docker hub:

I'm expecting to be able to use the DOCKER_TAG within a non custom build and it seems to be empty.
My Dockerfile code is something like:

ARG DOCKER_TAG=v9.9.0
...
RUN if [ "$DOCKER_TAG" = "latest" ] || [ "$DOCKER_TAG" = "" ] ; then \
    echo "Building latest: $DOCKER_TAG" && dotnet build; \
    else \
    echo "Building version: ${DOCKER_TAG}" && dotnet build -p:"Version=${DOCKER_TAG:1};AssemblyVersion=${DOCKER_TAG:1}"; \
    fi

It seems that in both builds the DOCKER_TAG is empty...
Am I doing something wrong or is this a bug?

[andyneff]

It's been a while since I looked at this, but I believe DOCKER_TAG is an environment variable, not a "docker build arg".

By environment variable, I mean it is an environment variable set in the shell that calls docker build. As you may know, environment variables are not just "passed along" to docker build, you have to manually specify what gets passed along by using "docker build args" that are usually set to hard coded values, or variables/environment variables.

In order to manually pass along an environment variable like DOCKER_TAG you need to write your own hooks/build script, like in the example here

So instead of saying:

docker build -t $IMAGE_NAME .

You would want something like

docker build --build-arg=DOCKER_TAG="${DOCKER_TAG}" -t $IMAGE_NAME .

Only then will the environment variable DOCKER_TAG be passed along as the build arg variable DOCKER_TAG

Note: --build-arg=FOO="${BAR}" would pass the environment variable BAR into the docker build arg variable named FOO
Note: --build-arg=FOO="stuff" would set the docker build arg variable named FOO to the value "stuff"

I think this feature request is about adding the ability to specify a value from the webpage dockerhub and have it passed along somehow as a build arg. So the desire of this feature request is to set "stuff" from the webpage and have it set to the docker build arg "FOO", without needing to write your own hooks/build script, which is little different from what you are after.

For now @HarelM , you should be able to accomplish what you want specifically by using a hooks/build script example I gave you.

[HarelM]

@andyneff Thanks for the info!
Well... in that case I rather just use my CI to build and publish the docker image instead of trying to understand another partial CI environment which needs all kind of hooks and what not.
I would expect these exiting environment variables to be passed to the docker image build flow and that not everyone would need to create a custom build hook for this feature, but I guess I'm in the minority here... :-)
I understand there's a workaround but seeing that a lot of people need to use this workaround, and it feels like it should be part of the build system, again, in my Opinion...

[gpaluk]

I'm laughing because DockerHub... You had one job! Really?.. Why are the key value pair fields sat on the DockerHub auto-build configuration page? This has been open since 2015 and people are confused between setting those key value pairs and the notion of adding build variables via --build-args. I'm lost for words here tbh. What's more adding the build-args will force the defaults that DockerHub set thus losing flags such as caching. Is there documentation that shows each of the flags and its corresponding value such that they can be re-added to any build-args value that are passed via the `hooks/build.sh' file. P.S. Downvoters cannot handle the truth! (y)