Unable to analyze the image using dive

[venkatn087]

What happened: I am using RHEL9 free tier from AWS and installed the dive with below steps by referring the github page https://github.com/wagoodman/dive and the version is dive 0.11.0
i have downloaded the Docker image "nginx:latest" by using "docker pull nginx" command
then i have executed "dive nginx:latest", getting below error

[ec2-user@ip-172-31-36-53 ~]$ sudo dive nginx
Image Source: docker://nginx
Fetching image... (this can take a while for large images)
cannot fetch image
could not find image config
[ec2-user@ip-172-31-36-53 ~]$

Could you please let me know how to scan an image using dive?

Anything else we need to know?: I am interested to know how to include the dive into jenkins CI/CD pipeline. like we are using Free style jobs so need to include the configuration in the "execute shell"

Environment: RHEL9

• OS version:
  - [ec2-user@ip-172-31-36-53 ~]$ uname -a
  Linux ip-172-31-36-53.ec2.internal 5.14.0-362.8.1.el9_3.x86_64 List layers #1 SMP PREEMPT_DYNAMIC Tue Oct 3 11:12:36 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
  [ec2-user@ip-172-31-36-53 ~]$

• Docker version (if applicable)
  - [ec2-user@ip-172-31-36-53 ~]$ docker --version
  Docker version 25.0.0, build e758fe5
  [ec2-user@ip-172-31-36-53 ~]$

[zerok]

Could it be that this is related to a Docker update that was released last weekend? I also see the same issue on Docker v25.0.0 but haven't yet seen in on v24.0.7.

[mark2185]

@rajiv-k is this something you could look into?

[tbroyer]

Apparently, the files no longer follow the same naming logic, so this code no longer detects the config file, and this one later fails.

Here's the same image (same Dockerfile) built with Docker 24.0.7 (pulled from our registry last week):

$ tar tf redacted.tar 
41b7e5a676314cb0e6b37013986b3abc3175bd1711e54574baf2619b56e00d4b/
41b7e5a676314cb0e6b37013986b3abc3175bd1711e54574baf2619b56e00d4b/VERSION
41b7e5a676314cb0e6b37013986b3abc3175bd1711e54574baf2619b56e00d4b/json
41b7e5a676314cb0e6b37013986b3abc3175bd1711e54574baf2619b56e00d4b/layer.tar
8d179503ce3761ab615a6c26c87cb707d9e622b5862ee09e3ee020895b06d545.json
e2745f779510e653aa2510f80e46e95a450d990ba58d40439e2a173dd389dbc8/
e2745f779510e653aa2510f80e46e95a450d990ba58d40439e2a173dd389dbc8/VERSION
e2745f779510e653aa2510f80e46e95a450d990ba58d40439e2a173dd389dbc8/json
e2745f779510e653aa2510f80e46e95a450d990ba58d40439e2a173dd389dbc8/layer.tar
ec777bf3518453637ef782f8888a75ac522f0c6aaf033ee3a26214d1f6ce463f/
ec777bf3518453637ef782f8888a75ac522f0c6aaf033ee3a26214d1f6ce463f/VERSION
ec777bf3518453637ef782f8888a75ac522f0c6aaf033ee3a26214d1f6ce463f/json
ec777bf3518453637ef782f8888a75ac522f0c6aaf033ee3a26214d1f6ce463f/layer.tar
manifest.json
repositories

and saved from a failed CI build today, with Docker 25.0.0:

$ tar tf xxx.tar 
blobs/
blobs/sha256/
blobs/sha256/07f6577589fb824de3bf45f2f7954fcf108bae5a6fc3b44e16029729733c454b
blobs/sha256/4dce6976862b95892955b4bd419eeff40d30fbe21fd1bbcefde6bf8b3e9491e0
blobs/sha256/583c2b89a34971925363d1f3de987bb19c907b98f47b808beb8954f699943913
blobs/sha256/8e87ff28f1b5ff2d5131999ccfa1e674cb252631c50683f5ee43fad59cbea8e1
blobs/sha256/9c7350069c7fb5363a35bbf8217870ee7d0f6cc839d330c64a4d13a306edcd8c
blobs/sha256/a08f2f0d2d5e0ed1f1b74bb62a504ff68cf3ea7f803060a0ca1340109bbc99d0
blobs/sha256/cc52537ba04fe1c94df2117aa105dfa740287df3069206ce6c95873dd5b7d350
blobs/sha256/db7eb0536bb70e202a19fef88169b53b656eeba01899f3f4dcb8f5dd7191e6ce
index.json
manifest.json
oci-layout
repositories

In the first case, the config file is that 8d179503ce3761ab615a6c26c87cb707d9e622b5862ee09e3ee020895b06d545.json at the root of the archive, in the second, it's the blobs/sha256/a08f2f0d2d5e0ed1f1b74bb62a504ff68cf3ea7f803060a0ca1340109bbc99d0 file. Note how we can no longer tell just from their name which files are JSON and which are TAR files.

This is I believe because:

The docker image save tarball output is now OCI compliant. moby/moby#44598

(https://docs.docker.com/engine/release-notes/25.0/)

[Borph]

Same here using dive 0.11.0 as docker image, Docker version 25.0.0, build e758fe5, Ubuntu

[mark2185]

@tbroyer since you've already dived into the relevant code, are you interested in sending a PR?

[tbroyer]

@mark2185 I started looking at it yesterday, but am really not sure how to approach it: start reading each file in the tar looking for a "magic number"? (knowing that tar and json don't really have such things) and/or trying to load them as tar or json and ignoring errors? (this could imply buffering a big chunk of the file) should this new/fallback approach be limited to entries in blobs/? Also, I haven't written Go in years.

I'll happily review a PR and test it though, and could help brainstorm the best approach.

[mark2185]

Fair enough, wanted to be sure before taking a crack at it.

I'd wager a good way would be initializing a tar.NewReader with the buffer and invoking Next(), if there's an error (that's not a io.EOF) we're definitely not reading a tar and hopefully we're reading a JSON.

Then we try unmarshalling it to see how well that works out, and if that errors out as well we can safely say what the hell.

[tbroyer]

You have to take gzip into account too, and remember that in many (most?) cases we're reading a tar that's directly streamed from the Docker daemon, and is not seekable. This means adding some buffering; 512 bytes should be enough as that's the size of a tar entry header (according to https://en.wikipedia.org/wiki/Tar_(computing)#File_format), but what if it's gzipped? maybe in this case it can be assumed to not be JSON so the fallback to JSON, and need for seeking back to the start of the entry, is not needed?

That would mean doing something like:

for each tar entry:
  // use the current algorithm based on file names, then do the following:
  else: // handle OCI-compatible Docker images
    buffer = read 512 bytes
    if buffer matches gzip magic number:
      create gzip stream (from a MultiReader on the buffer and remaining of the tar entry)
      create tar reader
      try processing layer, continue to next tar entry on error (ignoring it)
    else:
      create a tar reader (from a MultiReader on the buffer and remaining of the tar entry)
      try processing layer
      on error, try processing JSON (from a MultiReader on the buffer and remaining of the tar entry; assuming/hoping the previous tar reader didn't consume past the buffer)

[mark2185]

@tbroyer if I'm reading the image layout specification correctly, the index.json should have the mediaType indicating which type it is.

Can't check because I don't have such an image, which leads me to my next question - where did you get docker v25?
I have v24.0.7 on archlinux and a couple of friends I asked (windows, macOS) don't have v25 either.

[tbroyer]

@tbroyer if I'm reading the image layout specification correctly, the index.json should have the mediaType indicating which type it is.

Yes, but nothing guarantees that you'll see the index.json tar entry before the others, so either you possibly process the tar twice (once to get the index –you could also get the manifest.json instead– and then again looking up each tar entry in the index/manifest to know how to process it …but that means storing the tar to disk temporarily when resolving it from the Docker daemon), or you "content sniff" and parse/process each tar entry (like the pseudo algorithm above)
Maybe extracting the index or manifest first would be the best way forward, but that's a major change.

Can't check because I don't have such an image, which leads me to my next question - where did you get docker v25? I have v24.0.7 on archlinux and a couple of friends I asked (windows, macOS) don't have v25 either.

I too am on Arch so still on 24.0.7, our CI is on Ubuntu though, and we get the Docker Engine packages straight from Docker, Inc.: https://docs.docker.com/engine/install/ubuntu/

[tbroyer]

I managed to generate the .data/test-docker-image.tar on our CI server using make generate-test-data (I created an empty .scripts/ so it's not exactly equivalent to the TAR that's checked into the repository)
Note that I had to ZIP the TAR for GitHub to accept the upload 🤷‍♂️
test-docker-image.zip

And I reproduce the issue with it:

$ dive --source docker-archive test-docker-image.tar 
Image Source: docker-archive://test-docker-image.tar
Fetching image... (this can take a while for large images)
cannot fetch image
could not find image config

[utamas]

I'm on ubuntu 22.04 using docker 25.0.0 and run into this problem. Can I help in any way?

[saderi]

I have same problem

$ dive ubuntu:22.04
Image Source: docker://ubuntu:22.04
Fetching image... (this can take a while for large images)
Handler not available locally. Trying to pull 'ubuntu:22.04'...
22.04: Pulling from library/ubuntu
29202e855b20: Pull complete 
Digest: sha256:e6173d4dc55e76b87c4af8db8821b1feae4146dd47341e4d431118c7dd060a74
Status: Downloaded newer image for ubuntu:22.04
docker.io/library/ubuntu:22.04
cannot fetch image
could not find image config

System info:

Debian GNU/Linux 12
Docker version 25.0.1, build 29cf629
dive 0.11.0

[tbroyer]

I spent a couple hours and managed to get something working (see PR #500)

Tested on the included test-oci-docker-image.tar (same as test-docker-image.tar above) and on a real image built on our CI.

$ tar tf .data/test-oci-docker-image.tar 
blobs/
blobs/sha256/
blobs/sha256/014519b01ca15d1f716f2f53f874f36d1bff95eb1f2428b5dcb33b9c485c1ff7
blobs/sha256/0829d7d7c7533dfc6621a33eefcadfd6a0bb46aee864707604c1c946247cfc66
blobs/sha256/112d5a79660b130165194e86eb2d2d525724193d19f38a98139ff5dbd82e1b42
blobs/sha256/14e14481ff1d80a1052caa3870088a9fc12daa38f7fcc90e909efdfacffa3975
blobs/sha256/234c2d3f7286a73740b75a9177daa92527c5bca1fb8503f36c59a6110400d889
blobs/sha256/2e112031b4b923a873c8b3d685d48037e4d5ccd967b658743d93a6e56c3064b9
blobs/sha256/2f7025a8dd04f290d91ff47e1eedba90d76b1f58770e252b217eddd256933448
blobs/sha256/3031c0509e8328c76699618109bf52fc7de544c48a7c81901f4dc4cfef64a790
blobs/sha256/3806a7198fa7bc58ee31957786b96d643a77005a1fcd47680cbeced5dd4ceabf
blobs/sha256/4280bee4bc4cc547c1018e6c4aa6208dbecf67cf94c022ae155d67edf10ac32e
blobs/sha256/600e40802fefb0fcb5e2ba95b22cf9cf71b5182b12f96bb59cbc1b8fe4581dd4
blobs/sha256/6ed8c8c6b87a129d5923c3ffc069dc30388e52d51b94b96c071e0acd800a470e
blobs/sha256/730cfffc47b3ea3bfea77e753020ed67e5b787f39cad4b99c8c6dcea4398dbaf
blobs/sha256/74ff67166bdc723d124c08c6bb910c4f1c926846c3697fd96508e646230ca6a6
blobs/sha256/7e683cc329ff4e9a4d6dca80afd8490386be2c819be32de399de1808d4688d36
blobs/sha256/8e522620ef6a866882a1e2f2839c14d84c5cba5196db6783c334859e63ade4cc
blobs/sha256/93880e455d5aaa2af218dbcdd730dd6bd98743227738fdc64af837a768fa5ec3
blobs/sha256/a13e34bacfb27d2997e7dd941d8c989c44fe60e44afb7ffc73a0f2ef0fdda515
blobs/sha256/ac5c23ff42cb8be0b34b800bc2666a130ae6d853747d00fe4f8c17747432fdf4
blobs/sha256/ade2ae22142b7189be75aed479b9d06219a08652c11364e9c210aedebff7cd20
blobs/sha256/b5233acb40ee809c25225c27adc564291c88e6568401482d31c7530f0adf7bf5
blobs/sha256/bcd0d0bb27f99ff222277301a9fef50783b8ce911c40caa43c73491c472b562b
blobs/sha256/c135c36ce966c7b1ddbe492d3aed61b9a524e0778cd96a46e61d8e0a569cd9ae
blobs/sha256/c6e552ba03945bd8a68c68d7c65cbd15b553ca4a5caf17731718cb41994c4283
blobs/sha256/cbf85c261cfa45049e0ca75a715c4665a3c9debc22857187d6a3b70abab516dc
blobs/sha256/d19b18d2a66dda9837f3b14de66ef706697dde765edb2695fd8fbb64a0b6eb4e
blobs/sha256/d342c122b31c266c00c5bfa7d26afc7add7894c01d8ca136936725dc7127b0b3
blobs/sha256/e299237efeab238348e0af2d648f86a61344561d4f1a9df00f38802686ead2c4
blobs/sha256/e405d8993d37d5b3af9faeacb79d51ce46ec8be06ce746b82a576cac8c5878e4
blobs/sha256/fff04f37b214376b55e53cf8cb253908f8192e7efa31c8647867f67f316fe045
index.json
manifest.json
oci-layout
repositories
$ ./dive_linux_amd64 --ci docker-archive://.data/test-oci-docker-image.tar 
  Using default CI config
Image Source: docker-archive://.data/test-oci-docker-image.tar
Fetching image... (this can take a while for large images)
Analyzing image...
  efficiency: 99.3041 %
  wastedBytes: 50845 bytes (51 kB)
  userWastedPercent: 50.0000 %
Inefficient Files:
Count  Wasted Space  File Path
    2         20 kB  /root/saved.txt
    2         20 kB  /root/example/somefile1.txt
    2         10 kB  /root/example/somefile3.txt
   11           0 B  /etc
Results:
  FAIL: highestUserWastedPercent: too many bytes wasted, relative to the user bytes added (%-user-wasted-bytes=0.5 > threshold=0.1)
  SKIP: highestWastedBytes: rule disabled
  PASS: lowestEfficiency
Result:FAIL [Total:3] [Passed:1] [Failed:1] [Warn:0] [Skipped:1]

[james-johnston-thumbtack]

The PR from @tbroyer fixes the problem for me. Until it gets merged and a new dive release made, here's the TL;DR of how I was able to get it compiled and running:

1. Prerequisite: I already have Go version 1.19.5 installed on my OS X host, so I went with that.
2. Commands for cloning and building:

git clone https://github.com/tbroyer/dive
cd dive
git checkout docker25compat
go build -o dive-tool

# Run the tool:
./dive-tool <your image name>

There are probably proper Makefile targets I should be using, but I didn't bother digging in to figure it out. This worked.