Feedback thread: customDatasources
#23286
Replies: 18 comments 36 replies
-
I love the feature! I've already been experimenting with this and it proves to be really useful. One thing, as far as I understand the implementation (see the remaining TODO), it expects the custom registry to return JSON document. In one of my use-cases, the API returns a plain text string that is supposed to represent latest version (GET on /custom api return body "1.0.0" or "123.123.asd-123.zxcase" or even "my-custom-versioning-scheme") In that case, parsing it fails as it's not a valid JSON (I guess because the body is unquoted so it's not treated as string literal). Is that something that can be worked around at a current state? Maybe we could have some kind of configuration option as to how to translate whatever comes from API to JSON literal? |
Beta Was this translation helpful? Give feedback.
-
Idea: allow to run to external binaries to fetch data. The output is then fed into the current logic with Jsonata rules. My use case here is |
Beta Was this translation helpful? Give feedback.
-
I'm new to renovate, I love it, it love the docs, but I feel there's a lot of concepts being thrown at me - I think I would really benefit from this custom data source, so some documentation more specific to my type of use case would help. I feel I just need some url generator, a json parser for the content - its a bit hard to know which of the matchers and templates are relevant. |
Beta Was this translation helpful? Give feedback.
-
I really like it. I'm using it to get releases of Hashicorp tools using their API and update Ansible playbooks. |
Beta Was this translation helpful? Give feedback.
-
I used the custom datasource to extract supported kubernetes versions from Gardener and that worked great. Big thanks for adding this! Especially combining this as well with hostRules for authentication did work fully as expected. Only thing, which did cost me some time and might benefit from improving is the documentation about the Here e.g. the configuration we use to track available and supported kubernetes versions in Gardener: const config = {
hostRules: [
{
matchHost: "https://api.gardener.company.com",
token: "<gardenerToken>",
}
],
customDatasources: {
"gardener-k8s-supported": {
defaultRegistryUrlTemplate: 'https://api.gardener.company.com/apis/core.gardener.cloud/v1beta1/cloudprofiles/aws',
transformTemplates: [
`{"releases": $$.spec.kubernetes.versions[classification = 'supported'].{"version": $.version, "sourceUrl":"https://github.com/kubernetes/kubernetes", "changelogUrl":$join(["https://github.com/kubernetes/kubernetes/releases/tag/",$.version])},"sourceUrl": "https://github.com/kubernetes/kubernetes","homepage": "https://github.com/kubernetes/kubernetes"}`
],
},
"gardener-k8s-available": {
defaultRegistryUrlTemplate: 'https://api.gardener.company.com/apis/core.gardener.cloud/v1beta1/cloudprofiles/aws',
transformTemplates: [
`{"releases": $$.spec.kubernetes.versions.{"version": $.version, "sourceUrl":"https://github.com/kubernetes/kubernetes", "changelogUrl":$join(["https://github.com/kubernetes/kubernetes/releases/tag/",$.version])},"sourceUrl": "https://github.com/kubernetes/kubernetes","homepage": "https://github.com/kubernetes/kubernetes"}`
],
},
}
} |
Beta Was this translation helpful? Give feedback.
-
Great feature. I also love that it works with host rules which does not seem to be documented. That enables i.e. fetching data from the gitlab api. |
Beta Was this translation helpful? Give feedback.
-
This feature is useful for me to lock aws-for-fluent-bit to stable. https://github.com/aws/aws-for-fluent-bit#guidance-on-consuming-versions
{
"packageRules": [
{
"matchDepNames": ["public.ecr.aws/aws-observability/aws-for-fluent-bit"],
"matchDatasources": ["docker"],
"matchManagers": ["helm-values"],
"enabled": false
}
],
"regexManagers": [
{
"customType": "regex",
"fileMatch": [
"values\\.yaml"
],
"matchStrings": [
"public\\.ecr\\.aws\\/aws-observability\\/aws-for-fluent-bit:(?<currentValue>.*)"
],
"datasourceTemplate": "custom.aws-for-fluent-bit-stable",
"depNameTemplate": "public.ecr.aws/aws-observability/aws-for-fluent-bit"
}
],
"customDatasources": {
"aws-for-fluent-bit-stable": {
"defaultRegistryUrlTemplate": "https://raw.githubusercontent.com/aws/aws-for-fluent-bit/mainline/AWS_FOR_FLUENT_BIT_STABLE_VERSION",
"format": "plain"
}
}
} |
Beta Was this translation helpful? Give feedback.
-
Is there a way to debug your datasources? I tried pulling in CoreOS updates via their stream JSON APIs. I validated the transformers against the data in https://try.jsonata.org/, but still Renovate insists that the currently used version is the latest one. "regexManagers": [
{
"fileMatch": [
"template-x86_64.pkr.hcl"
],
"matchStrings": [
"coreos_release = \"(?<currentValue>\\S+)\""
],
"depNameTemplate": "coreos",
"datasourceTemplate": "custom.coreos-stable"
}
],
"customDatasources": {
"coreos-stable": {
"defaultRegistryUrlTemplate": "https://builds.coreos.fedoraproject.org/streams/stable.json",
"transformTemplates": [
"{\"releases\": [{\"version\": $$.(architectures.x86_64.artifacts.metal.release),\"changelogUrl\": \"https://fedoraproject.org/coreos/release-notes/?arch=x86_64&stream=stable\"}]}"
]
}
} |
Beta Was this translation helpful? Give feedback.
-
Very glad for this feature! Thank you! |
Beta Was this translation helpful? Give feedback.
-
I've returned and managed to get a lot further. A few things on my wish list:
To give an example, my latest struggle: All I had was: "Failed to look up custom.X package Y". |
Beta Was this translation helpful? Give feedback.
-
An Odyssey: Trying To Use Custom Datasources To Query APT-RepositoriesSo, we have two repositories on GitLab:
Renovate works perfectly fine for a large portion of our dependencies. We can update Docker image tags in Dockerfiles, Gitlab CI files, and Helm values.yml files. We can update Helm chart versions, Pipenv dependencies, and Terraform and Terragrunt dependencies. Great experience all around. However, we also had to setup a {
customType: "regex",
fileMatch: [
"(^|/|\\.)([Dd]ocker|[Cc]ontainer)file$",
"(^|/)([Dd]ocker|[Cc]ontainer)file[^/]*$"
],
matchStrings: [
"#[ \\t]+Renovate:[ \\t]+datasource=(?<datasource>[.-_a-z]+?)[ \\t]+depName=(?<depName>\\S+?)(?:[ \\t]+(lookupName|packageName)=(?<packageName>\\S+?))?(?:[ \\t]+versioning=(?<versioning>\\S+?|regex:\\/.*?\\/))?(?:[ \\t]+extractVersion=(?<extractVersion>\\S+?))?(?:[ \\t]+registryUrl=(?<registryUrl>\\S+?))?(\\r|\\n|\\r\\n)(?:ENV|ARG)[ \\t]+.+?_VERSION[ =]\"?(?<currentValue>\\S+?)\""
],
versioningTemplate: "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
} This is basically the example from the docs with some adjustments. I.e., it allows for something like # Renovate: datasource=repology depName=ubuntu_22_04/git
ENV GIT_VERSION="2.34.1" in our Dockerfile. Again, where we can use built-in datasources, this works fine. Tools from custom APT repositories - Approach 1Here's the issue: We build a Docker image that contains tools which are installed from custom APT repositories (e.g., So my first workaround was this: Renovate can use GitHub releases and Git tags as a datasource. All the tools we install are available from accessible repositories so we can examine them and use the release names / Git tags as versions. This didn't work because the names of the releases / tags don't correspond directly to the version notation used in the APT repositories. We tried to map versions (which worked but was very messy) but then you run into the issue that versions published on GitHub take a while until they show up in the APT repositories. So Repology would propose an updated Dockerfile which then couldn't be built because the version wasn't present in the APT repository.
Tools from custom APT repositories - Approach 2So, since Renovate doesn't have such a datasource, I tried using the Right, so my approach is this:
Here come the issues with this approach: The only template variable that we have access to in # Renovate: datasource=custom.local_apt_versions_file depName=google-cloud packageName=https://packages.cloud.google.com/apt/cloud-sdk/main/google-cloud-cli versioning=deb
ENV GCLOUD_VERSION="462.0.1-0" Of course, the
Now, Renovate and But lets run with it, everything after the
So I put my files into the base folder, I verified that the file URL was correct given how it was handled, ran Renovate and... nothing. The files were gone. Well, turns out that Renovate will
Tools from custom APT repositories - Approach 3So we can't use Of course, accessing these artifacts requires an authentication header. Renovate already has a {
matchHost: `${process.env.RENOVATE_ENDPOINT}/projects/${process.env.PROJECT_ID}/jobs/${process.env.APT_VERSIONS_JOB_ID}/artifacts/${process.env.APT_VERSIONS_FOLDER}`,
token: `${process.env.TOKEN}`,
authType: "Bearer"
} I run Renovate and... nothing. I get a 404 for the URL which in this case indicates an invalid authentication header. If the header had the correct format but the token was wrong, I would get 401. By the way, the trace log shows "Adding Bearer authentication for URL" but this is not done.
Finally, I tried to work around this issue by adding the authentication header myself. I added allowedHeaders: [
"Authorization",
"X-*"
] and to the headers: {
Authorization: `Bearer ${process.env.TOKEN}`
} This workaround also doesn't work. I don't know why, I stopped investigating due to time constraints at this point. I think it's because the repository we are working with is also on GitLab and Renovate internally adds hostRules to handle authentication. The
Conclusion / TLDRI'd appreciate if you would take the time to look at this @secustor and tell me where / if I went wrong here. For now, we can't use Renovate to update APT backed dependencies. I think my approach should be doable but I am blocked by the issues that I described. My proposals for remediation are quoted. |
Beta Was this translation helpful? Give feedback.
-
Hi there, Please do not unnecessarily For example, never It's OK to comment in an issue or discussion after multiple days or weeks. But please, still don't Thanks, the Renovate team |
Beta Was this translation helpful? Give feedback.
This comment has been minimized.
This comment has been minimized.
-
Found an additional interesting use case for k3s https://github.com/visualon/renovate-config/pull/10/files |
Beta Was this translation helpful? Give feedback.
-
Custom datasources have been very helpful to us. Some general remarks:
Some of our usecases:
|
Beta Was this translation helpful? Give feedback.
-
Thanks for the feature, it is great and easy to use but hard to debug : while I did success with some HTML page, I am trying to process release at sqlite (https://sqlite.org/download.html) using a customDatasources with format=html. That's would be cool if we could:
Here is the JSON I am "expecting" and the "result" I am trying to feed renovate: https://try.jsonata.org/vyDcg2vEt In this case, Renovate fails to update the version and it is hard to determine the culprit here. I also tried github-tags / releases as source and got the same kind of result for other dependency, such as libexpat. |
Beta Was this translation helpful? Give feedback.
-
I'll retry with new line because https://json5.org/ says that Strings may span multiple lines by escaping new line characters. I did try with new line character, but not with
I'll give it a try tomorrow, thanks 👍
In case it fails again (I did try in private mode, ...); the JSON I am expecting is something like this: {
"releases": [
{ "version": "https://sqlite.org/2024/sqlite-amalgamation-3460000.zip" },
{ "version": "https://sqlite.org/2024/sqlite-autoconf-3460000.tar.gz" },
...
]
} The JSON data is only the result of executing this javascript in developer console on https://sqlite.org/download.html (I removed links not referencing archive). console.log(JSON.stringify({releases: Array.from(document.querySelectorAll('a[href]'))
.map(anchor => anchor.href)
.filter(href => href.startsWith('http://') || href.startsWith('https://'))
.filter(href => href.match(/\.(zip|7z|tar\.(gz|bz|bz2))$/i ))
.map(href => ({version: href})) })) |
Beta Was this translation helpful? Give feedback.
-
Ok, no wonder it never worked: the page is using javascript to generate the links whereas a HTML comment contains the version in CSV. I'll have to resort to plain text + jsonata. |
Beta Was this translation helpful? Give feedback.
-
Tell us more.
Tell us how you like the new
customDatasources
experimental feature. What works for you, what doesn't work for you? Do you see anything we could improve?Let us know what you think! 😉
Beta Was this translation helpful? Give feedback.
All reactions