-
Notifications
You must be signed in to change notification settings - Fork 5.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docker-compose build and docker build lead to different IDs #883
Comments
I suspect this is the same issue as #651 You could try with master, or the latest docker-compose==1.1.0-rc1 |
Still an issue with
Weird ... same until
|
Could it make a difference that compose is using a different version of the API then the docker client? Do you make use of a |
How can I check the version of both?
btw: Testing is based on these docs... This is the .dockerignore: |
Compose uses the same docker version, but pins the API version to an older version (e.g. /API/v14), to stay compatible with older versions of Docker. Sometimes that could result in slightly different behavior than using the docker client directly, which will use the latest version of the API. Thanks for the extra info! |
I also experiment the same thing. fig build vs docker build yields to different images id. The issue seems to be with COPY. While copying the same file, the file checksum is not the same, yielding in different image ID from that step on:
Docker:
Docker version 1.3.2, build 39fa2fa/1.3.2 |
Also checked what the difference is; an empty file, a Dockerfile and a fig.yml; touch testfile
cat << EOF > Dockerfile
FROM scratch
ADD ./testfile /a-file
COPY ./testfile /b-file
EOF
cat << EOF > fig.yml
app:
build: .
EOF Build with Fig and check the docker logs; $ fig build
INFO[524009] POST /v1.12/build?q=False&rm=True&t=figtest_app&nocache=False
INFO[524009] +job build()
INFO[524009] -job build() = OK (0) And with docker-1.3.3 (just the client, the daemon is 1.5.0-rc4) ./docker-1.3.3 build -t footest_app .
INFO[525891] POST /v1.15/build?rm=1&t=footest_app
INFO[525891] +job build()
INFO[525893] -job build() = OK (0) So differences are;
Not sure if these would make a difference? Are the files sent different by Fig? |
I believe the tarball of the files is created by the client |
Different file ordering in the tarball maybe? |
I tested this using Fig 1.0.1, not yet with Compose, so it might have been fixed. My example above should almost be copy-pasta-ble, so easy to check |
@aanand; but isn't the hash calculated per-file? Does order in the tar matter? |
@thaJeztah correct, my bad. I've investigated your example a little, using fig checked out at the My process:
Two things I've noticed:
|
@aanand hm, you may be right wrt the image-ids, I'll need to check. However, what's strange is that (not behind my computer now) they didn't use the same cache. Ie; building twice with Fig, the second Fig build would use the cache, but doing a build with Docker after that didn't use the cache. At this moment, I see it merely an "inconvenience", but would this become a problem if image signing/verification is taken into account? Would this lead to a different result? In the end, the same Dockerfile + build-context should lead to exactly the same image, regardless if Fig (Compose) or Docker was used to build that image. If not, that may be a problem. I'm not entirely sure I'm on the right path here wrt the signing process, so maybe @dmcgowan could shed some light. I'll have a look tomorrow if I can come up with some test scenario as well (Perhaps I'm overlooking something, just intrigued what would lead to the difference) |
@thaJeztah shouldn't have any effect on signing/verification. Right now the signatures occur on push and verification on pull and the content is frozen in between. The sign/verify content and new registry is still considered beta, but I don't see any potential problems there. Also worth noting that since these IDs are not content hashes, they will never generate the same ID unless its using cache even when the content is exactly the same. |
Thanks for kicking in, Derek. No reason to worry then! (and, yes, I'm fully aware signing is still a tech-preview). Still curious why Fig and Docker don't share each other's layer-cache in this case. Possibly because of a different TAR implementation? Will do some experimenting tomorrow to satisfy my curiosity :) |
Good to know - so hopefully all it's going to result in is slower builds in an edge case - but it's still bizarre and concerning that a file (of zero length!) is hashing differently between fig and docker. |
ah! I'm glad I found this thread. I'm seeing this, as we are using This slows the test process by many minutes, but it also produced different artifacts... so what was being tested was not the same as what was being deployed. as @thaJeztah noted, this is more of an inconvenience, but one that took awhile to track down and then explain to folks that yes they are different but they are also the same so everything is ok... and another, related oddity, is if I take the image from at the end of the day, I agree with @thaJeztah ; |
docker-py==1.6.0 I still affected by this inconsistency |
I think this is "expected", see #883 (comment). |
@dnephin I get the point on IDs not being the content hash, but why do we fail to get cache reuse when docker-py is just passing tar stream? See: moby/moby#18611 (comment) |
Tar implementations can vary, which may be what is causing cache misses when different clients are used. Docker 1.10 is supposed to be introducing content addressable layers, which may resolve this issue. |
docker will be using content addressable ids in 1.10 which should resolve this issue. |
This is still an issue. I'm running: I have replicated this with a single copy call of an empty file here: |
Looks like the changes in 1.10 did not fix this. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Does anyone know if the issue has been solved? |
This issue has been automatically marked as not stale anymore due to the recent activity. |
I am experiencing this issue. Here my versions:
|
Note: 1.25 will come with |
@ndeloof I just tried setting the environment variable you speak of, and now I can build the same image with THREE different hashes. docker build .
docker-compose up --build
COMPOSE_DOCKER_CLI_BUILD=true docker-compose up --build This results in 3 fully functional docker images, all with different hashes. I think it has something to do when using a I get cache hits for all of my Dockerfile steps until the first |
…er-py for building images The build cache sometimes works sometimes doesn't. The images pushed from the master branch were sometimes producing reusable layers, sometimes not. So the caching was working non-deterministically. The underlying issue is docker/compose#883 Closes #6802 from kszucs/docker-compose-cli Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
I see the same result as @JeremyLoy |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
This issue has been automatically closed because it had not recent activity during the stale period. |
This issue should be left open |
This is still an issue |
I created a new issue as this one is being ignored: #7905 |
…er-py for building images The build cache sometimes works sometimes doesn't. The images pushed from the master branch were sometimes producing reusable layers, sometimes not. So the caching was working non-deterministically. The underlying issue is docker/compose#883 Closes #6802 from kszucs/docker-compose-cli Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
The fillGo18FileTypeBits func was added in 1a451d9 to keep the tar headers consistent with headers created with go1.8 and older. go1.8 and older incorrectly preserved all file-mode bits, including file-type, instead of stripping those bits and only preserving the _permission_ bits, as defined in; - the GNU tar spec: https://www.gnu.org/software/tar/manual/html_node/Standard.html - and POSIX: http://pubs.opengroup.org/onlinepubs/009695399/basedefs/tar.h.html We decided at the time to copy the "wrong" behavior to prevent a cache-bust and to keep the archives identical, however: - It's not matching the standards, which causes differences between our tar implementation and the standard tar implementations, as well as implementations in other languages, such as Python (see docker/compose#883). - BuildKit does not implement this hack. - We don't _need_ this extra information (as it's already preserved in the type header; https://pkg.go.dev/archive/tar#pkg-constants In short; let's remove this hack. This reverts commit 1a451d9. This reverts commit 41eb61d. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The fillGo18FileTypeBits func was added in 1a451d9 to keep the tar headers consistent with headers created with go1.8 and older. go1.8 and older incorrectly preserved all file-mode bits, including file-type, instead of stripping those bits and only preserving the _permission_ bits, as defined in; - the GNU tar spec: https://www.gnu.org/software/tar/manual/html_node/Standard.html - and POSIX: http://pubs.opengroup.org/onlinepubs/009695399/basedefs/tar.h.html We decided at the time to copy the "wrong" behavior to prevent a cache-bust and to keep the archives identical, however: - It's not matching the standards, which causes differences between our tar implementation and the standard tar implementations, as well as implementations in other languages, such as Python (see docker/compose#883). - BuildKit does not implement this hack. - We don't _need_ this extra information (as it's already preserved in the type header; https://pkg.go.dev/archive/tar#pkg-constants In short; let's remove this hack. This reverts commit 1a451d9. This reverts commit 41eb61d. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The fillGo18FileTypeBits func was added in 1a451d9 to keep the tar headers consistent with headers created with go1.8 and older. go1.8 and older incorrectly preserved all file-mode bits, including file-type, instead of stripping those bits and only preserving the _permission_ bits, as defined in; - the GNU tar spec: https://www.gnu.org/software/tar/manual/html_node/Standard.html - and POSIX: http://pubs.opengroup.org/onlinepubs/009695399/basedefs/tar.h.html We decided at the time to copy the "wrong" behavior to prevent a cache-bust and to keep the archives identical, however: - It's not matching the standards, which causes differences between our tar implementation and the standard tar implementations, as well as implementations in other languages, such as Python (see docker/compose#883). - BuildKit does not implement this hack. - We don't _need_ this extra information (as it's already preserved in the type header; https://pkg.go.dev/archive/tar#pkg-constants In short; let's remove this hack. This reverts commit 1a451d9. This reverts commit 41eb61d. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When building containers for my project I noticed that, even if I build from the same Dockerfile,
fig
anddocker
build different image IDs in the end.I also tried to use the same tag, but the images still were different.
Built with fig:
Built with docker:
ec26eedffc59
and8fecc44750ee
is the same file, but added with different IDs.Is there anything I've overlooked or is this a bug?
Using fig 1.0.1 docker 1.3.2.
The text was updated successfully, but these errors were encountered: