Renovate getting the wrong latest tag with the docker datasource

[eyalzek]

What Renovate type, platform and version are you using?

hosted app on Github

Describe the bug

renovate uses the wrong tag in the getLabels request for the image: quay.io/bitnami/sealed-secrets-controller.

Relevant debug logs

DEBUG: getLabels(https://quay.io, bitnami/sealed-secrets-controller, build-683905581-683905588-linux-arm) (repository=eyalzek/renovate-test)

Have you created a minimal reproduction repository?

reproduction is done here (the relevant image is quay.io/bitnami/sealed-secrets-controller) https://github.com/eyalzek/renovate-test

Additional context

it's hard to test this, but I assume the problem is with the following line:

renovate/lib/datasource/docker/index.ts

Line 640 in f8580a1

const latestTag = tags.includes('latest') ? 'latest' : tags[tags.length - 1];

because the registry for this image does actually include a latest - although it's out of date, it still gets the last element from the tags array. Though I have no idea why the last element comes out to be build-683905581-683905588-linux-arm, browsing through the registry it seems like tags are ordered properly (https://quay.io/repository/bitnami/sealed-secrets-controller?tab=tags).

[github-actions[bot]]

Thank you for providing a reproduction! 🎉 🚀

The Renovate team will take a look at the reproduction repository. Once we confirm the provided repository reproduces the problem, the label will be changed to reproduction:confirmed.

[rarkins]

The reproduction repository seems different from the description here. Can you please double check and clarify?

[eyalzek]

The reproduction repository seems different from the description here. Can you please double check and clarify?

It's not, it just had some additional stuff in there. I cleaned it up so it's more obvious.

[eyalzek]

why is the issue closed ??

[rarkins]

The default versioning that Renovate uses for Docker is quite flexible by necessity, because many packages like this don't restrict themselves to a proper tagging convention (e.g. semver). In this case b sorts higher than 0, so it is selected as the highest.

To solve this you can configure versioning to be semver and only semver-compatible versions will be selected:

{
  "packageRules": [{
    "matchPackageNames": ["quay.io/bitnami/sealed-secrets-controller"],
    "versioning": "semver"
  }]
}

[eyalzek]

As I said, it's already done. I was asking if you somehow have access to the managed app dashboard, or can I somehow give you access to it?

[rarkins]

I don't see any Pull Requests. When you said "exactly the same results" I assumed you meant an incorrect PR. What do you mean?

[eyalzek]

the actual problem is that renovate assumes the wrong tag is the latest tag, in this case build-683905581-683905588-linux-arm and therefore it never opens a PR to update quay.io/bitnami/sealed-secrets-controller:v0.12.5 to any newer version... The expected result would be a PR to update the image to quay.io/bitnami/sealed-secrets-controller:v0.15.0

[rarkins]

Try this: eyalzek/renovate-test#1

[eyalzek]

still, same behavior:

DEBUG: getLabels(https://quay.io, bitnami/sealed-secrets-controller, build-683905581-683905588-linux-arm)
DEBUG: Package releases lookups complete
{
  "baseBranch": "master"
}
DEBUG: branchifyUpgrades
DEBUG: 0 flattened updates found:
DEBUG: Returning 0 branch(es)
DEBUG: config.repoIsOnboarded=true
DEBUG: packageFiles with updates
{
  "config": {
    "kubernetes": [
      {
        "packageFile": "deployment.yaml",
        "deps": [
          {
            "depName": "quay.io/bitnami/sealed-secrets-controller",
            "currentValue": "v0.12.5",
            "replaceString": "quay.io/bitnami/sealed-secrets-controller:v0.12.5",
            "autoReplaceStringTemplate": "{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}",
            "datasource": "docker",
            "depIndex": 0,
            "updates": [],
            "warnings": [],
            "fixedVersion": "v0.12.5"
          }
        ]
      }
    ]
  }
}

[viceice]

I think allowedversions needs to be sett to an regex.

  {
  ...
  "allowedversions": "/^v\\d+\\.\\d+\\.\\d+$/"
  }

[eyalzek]

shall we reopen my issue and move the discussion there if this is an actual issue?

[viceice]

nope, it's more a quay.io issue, so there should an issue

 curl -sv 'https://quay.io/api/v1/repository/bitnami/sealed-secrets-controller/tag/?limit=10&onlyActiveTags=true' | jq
*   Trying 3.213.173.170:443...
* TCP_NODELAY set
* Connected to quay.io (3.213.173.170) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [98 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2965 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=North Carolina; L=Raleigh; O=Red Hat, Inc.; CN=*.quay.io
*  start date: May 27 00:00:00 2020 GMT
*  expire date: Jun  1 12:00:00 2022 GMT
*  subjectAltName: host "quay.io" matched cert's "quay.io"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55b04bffb7c0)
} [5 bytes data]
> GET /api/v1/repository/bitnami/sealed-secrets-controller/tag/?limit=10&onlyActiveTags=true HTTP/2
> Host: quay.io
> user-agent: curl/7.68.0
> accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
} [5 bytes data]
< HTTP/2 200
< server: nginx/1.12.1
< date: Tue, 16 Mar 2021 15:19:28 GMT
< content-type: application/json
< content-length: 3680
< access-control-allow-origin: *
< access-control-allow-methods: HEAD, OPTIONS, GET
< access-control-max-age: 21600
< access-control-allow-headers: AUTHORIZATION, CONTENT-TYPE, X-REQUESTED-WITH
< x-frame-options: DENY
< strict-transport-security: max-age=63072000; preload
<
{ [3680 bytes data]
* Connection #0 to host quay.io left intact
{
  "has_additional": true,
  "page": 1,
  "tags": [
    {
      "name": "v0.15.0",
      "reversion": false,
      "start_ts": 1613748112,
      "last_modified": "Fri, 19 Feb 2021 15:21:52 -0000",
      "manifest_digest": "sha256:f9c749a3fb897f8599d9b1710ba2678b26264a33aa3c7a3b4291fcca3036e423",
      "is_manifest_list": true
    },
    {
      "name": "v0.15.0-759643132",
      "reversion": false,
      "start_ts": 1613747904,
      "last_modified": "Fri, 19 Feb 2021 15:18:24 -0000",
      "manifest_digest": "sha256:f9c749a3fb897f8599d9b1710ba2678b26264a33aa3c7a3b4291fcca3036e423",
      "is_manifest_list": true
    },
    {
      "name": "v0.15.0-759643132-linux-arm",
      "reversion": false,
      "start_ts": 1613747900,
      "image_id": "99bcecb65c61d69bc8debba33e32afc451f76f70077d37415d7ce5ee7e82c89b",
      "last_modified": "Fri, 19 Feb 2021 15:18:20 -0000",
      "manifest_digest": "sha256:b0729ddb748905523cb014c345d682e7ec9356d68990075d1ce777bc7bc3166e",
      "docker_image_id": "99bcecb65c61d69bc8debba33e32afc451f76f70077d37415d7ce5ee7e82c89b",
      "is_manifest_list": false,
      "size": 16545925
    },
    {
      "name": "v0.15.0-759643132-linux-arm64",
      "reversion": false,
      "start_ts": 1613747834,
      "image_id": "9d8f92cc5b2505abb32fc44931a706a2b7809e038bc118047da3853346621442",
      "last_modified": "Fri, 19 Feb 2021 15:17:14 -0000",
      "manifest_digest": "sha256:ec8b0e8896d76bf10aeb8a1b6973afa4d5183884be8e91911fc2273efd6d98ea",
      "docker_image_id": "9d8f92cc5b2505abb32fc44931a706a2b7809e038bc118047da3853346621442",
      "is_manifest_list": false,
      "size": 16413477
    },
    {
      "name": "v0.15.0-759643132-linux-amd64",
      "reversion": false,
      "start_ts": 1613747770,
      "image_id": "612e3dd28ada252750d2d95fa61434d5d286e7ddb01f395439569c2d20eb91f4",
      "last_modified": "Fri, 19 Feb 2021 15:16:10 -0000",
      "manifest_digest": "sha256:ce7fd544838c04eadf5f9ecbe6b339a88e54996c22b6c8c9a3b463fdcbabc8c5",
      "docker_image_id": "612e3dd28ada252750d2d95fa61434d5d286e7ddb01f395439569c2d20eb91f4",
      "is_manifest_list": false,
      "size": 17552566
    },
    {
      "name": "v0.14.1",
      "reversion": false,
      "start_ts": 1611604856,
      "last_modified": "Mon, 25 Jan 2021 20:00:56 -0000",
      "manifest_digest": "sha256:4b59dc6f9087f5787ebca9473e837b3b38c6e72185fdc14d6d4ce57dbe20b4dd",
      "is_manifest_list": true
    },
    {
      "name": "v0.14.1-756132800",
      "reversion": false,
      "start_ts": 1611604662,
      "last_modified": "Mon, 25 Jan 2021 19:57:42 -0000",
      "manifest_digest": "sha256:4b59dc6f9087f5787ebca9473e837b3b38c6e72185fdc14d6d4ce57dbe20b4dd",
      "is_manifest_list": true
    },
    {
      "name": "v0.14.1-756132800-linux-arm",
      "reversion": false,
      "start_ts": 1611604659,
      "image_id": "adbfd15ee42a03322b5163d3cb4aca291bc435d3520390ec894e9130cc6629a4",
      "last_modified": "Mon, 25 Jan 2021 19:57:39 -0000",
      "manifest_digest": "sha256:f7623afbfedd3aec4d1837ac27fdd58e3a9036ec82317f3ffc8684553e570368",
      "docker_image_id": "adbfd15ee42a03322b5163d3cb4aca291bc435d3520390ec894e9130cc6629a4",
      "is_manifest_list": false,
      "size": 16538822
    },
    {
      "name": "v0.14.1-756132800-linux-arm64",
      "reversion": false,
      "start_ts": 1611604599,
      "image_id": "70e9e0ae38c96bff702d8c01adc401784d1ef5452c6d744fcc026fd83d0e04e7",
      "last_modified": "Mon, 25 Jan 2021 19:56:39 -0000",
      "manifest_digest": "sha256:0793465acf76eba033ed7b6b65dddb583881abdd4d88a567467cd7cecc5bb09f",
      "docker_image_id": "70e9e0ae38c96bff702d8c01adc401784d1ef5452c6d744fcc026fd83d0e04e7",
      "is_manifest_list": false,
      "size": 16390472
    },
    {
      "name": "v0.14.1-756132800-linux-amd64",
      "reversion": false,
      "start_ts": 1611604538,
      "image_id": "29089f53978afacd12a92a5f8401d7a98a54cc9683ed5281675cacc7040c02c9",
      "last_modified": "Mon, 25 Jan 2021 19:55:38 -0000",
      "manifest_digest": "sha256:07ec3f47b11275c852920beb34197af066037090938f11aa47887be2af8cbf94",
      "docker_image_id": "29089f53978afacd12a92a5f8401d7a98a54cc9683ed5281675cacc7040c02c9",
      "is_manifest_list": false,
      "size": 17539358
    }
  ]
}

[eyalzek]

I've done some curl foo, so it seems they simply return tags in creation order asc and we only get the first 1000

is this a limitation within renovate or it's actually impossible to get all tags from their api?

[viceice]

renovate/lib/datasource/docker/index.ts

Lines 458 to 464 in 295d5ab

	do {
	const res = await http.getJson<{ tags: string[] }>(url, { headers });
	tags = tags.concat(res.body.tags);
	const linkHeader = parseLinkHeader(res.headers.link as string);
	url = linkHeader?.next ? URL.resolve(url, linkHeader.next.url) : null;
	page += 1;
	} while (url && page < 20);

Renovate currently only fetches first 20 pages.

@rarkins should we check returned tags count and adjust max page fetch to get more tags ? But that will fail some time in future again, when there are more tags as we fetch. 🤔

[eyalzek]

well as things stand now, any quay repo with > 1000 tags might not work, would it not make sense to let the loop run until url is undefined? The problem is the overhead of such a loop?

[eyalzek]

I opened a PR for removing the 20 page limit here:
#9173

let me know if this makes sense to you or if you would prefer to just bump the limit (or go another direction entirely).

[rarkins]

Sorry, the proposed change is dangerous and cannot be accepted as-is, so I have closed it. Please create a new feature request issue back in the repo for us to discuss solutions firest. What might be possible is adding an admin-only option (i.e. for self-hosted) to set either (a) a page limit, or (b) a total items limit.

[eyalzek]

fair enough

[eyalzek]

Sorry, the proposed change is dangerous and cannot be accepted as-is, so I have closed it. Please create a new feature request issue back in the repo for us to discuss solutions firest. What might be possible is adding an admin-only option (i.e. for self-hosted) to set either (a) a page limit, or (b) a total items limit.

This would have to be a self-hosted solution to avoid possible load on the managed app?

[rarkins]

Yes, and several other risks