customDatasources: allow to run external binaries to generate result

[secustor]

Describe the proposed change(s).

This feature is only intended for self hosted instances because of the security implications!
Also it should be not enabled by default!

There should be a new global config option for limiting the allowed commands: allowedCustomDatasourceCommands.
This should mimic allowedPostUpgradeCommands.

Further there should be a new option

{
  "customDatasources": {
    "foo": {
      "registryCommandTemplates": [
        "echo {\"foo\":\"bar\"}' > myRegistry.json",
        "cat myRegistry.json"
      ],
      "...": ""
    }
  }
}

This will run all commands of registryCommands serially and will capture the output ( stdout ) of all commands and concat it at the and.

The resulting string is then be processed depending on the chosen format

Describe why we need/want these change(s).

#23286 (comment)

Currently if using the custom datasource there is no way we can support every use case or API. A step to allow even more concepts would be to option to handover the gathering of the raw data to an external process.

An example would be fetching files from an network share and then handing over the contents to Renovate or running queries from CLI tools such as aws or az for the respective cloud providers.

[chasecaleb]

This would be great! I would be equally happy if renovate could read versions from a (JSON) file instead of running commands too, which would accomplish the same thing if you run the commands right before running renovate. I think this might even be preferable because it decouples renovate from the registry commands and it's easier and safer to implement for renovate.

I have a few use cases at work where I'm considering writing a lightweight service that runs commands like this and then wraps the result in a customDatasource-compatible API response, but it would be nice to skip all the indirection and extra work of writing and deploying a service. Amazon EKS addons is one example: there isn't a Renovate data source and they don't have a (documented) API for it, but you can use the aws CLI to output them.

[z0rc]

they don't have a (documented) API for it

They do: https://docs.aws.amazon.com/eks/latest/APIReference/API_DescribeAddonVersions.html

Side note, maybe EKS addons datasource can be implemented as native option? Just like https://docs.renovatebot.com/modules/datasource/aws-machine-image/. As currently this DescribeAddonVersions API require AWS auth, which can be a nuisance on its own.

[taraspos]

In case of someone looking to renovate EKS Addons, I published (a bit over-engineered, but viable) solution that works well for me with GitHub Actions self-hosted renovate:

• AWS EKS Add-ons Datasource #29465 (comment)