Custom Manager + Custom Datasource not being triggered

[Andres-Lu]

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

37.176.0

If you're self-hosting Renovate, select which platform you are using.

None

What is your question?

Hello!

We've been trying to configure Renovate on a given repository to update our custom Terraform modules to the latest version we've published. We've set up a customManager to match all files ending in .tf that looks like this:

"customManagers": [
        {
            "customType": "regex",
            "fileMatch": [
                "\\.tf$"
            ],
            "datasourceTemplate": "custom.modules",
            "matchStrings": [
                "^\\s+source\\s+=\\s+\"(?<url>https?:\\/\\/[^\\/]+\\/)(?<depName>[^\\/]+)\\/(?<currentValue>[^\"]+?)\"$"
            ]
       }
]

The expectation is that it'd be able to pick up something like this within a Terraform file:

module "loki" {
  source = "https://modules.my-company.org/eks-loki/5.1.0"

  name = "blah"

  loki_object_store = "s3"
  s3_bucket_region  = "us-east-1"
}

Where depName should be eks-loki, and currentValue should be 5.1.0.

For the custom Datasource, we should be looking at an internal API Endpoint (available from the cluster) that is configured like this:

"customDatasources": {
        "modules": {
            "defaultRegistryUrlTemplate": "https://api.my-company.org/code/module-version/{{packageName}}",
            "transformTemplates": [
                "{ \"releases\": [{ \"version\": $.results[*].releases[*].version, \"changelogUrl\": $.results[*].changelogUrl, \"sourceUrl\": $.results[*].sourceUrl }], \"homepage\": $.results[*].homepage }"
            ]
        }
    }

The endpoint returns a JSON object formatted like this:

{
  "request_time": 1699389229,
  "results": [
    {
      "releases": {
        "version": "5.6.0"
      },
      "name": "eks-loki",
      "changelogUrl": "https://changelog-url.com",
      "sourceUrl": "https://source-url.com",
      "homepage": "https://homepage-url.com"
    }
  ]
}

Currently running Renovate from an EKS cluster (deployed through Helm), since the API endpoint is only available within our internal VPC. It seems to be matching the .tf files correctly, and apparently extracting something, but it doesn't trigger any changes nor do I find any debug level logs pointing to an error or a misconfiguration

DEBUG: manager extract durations (ms) (repository=my-tf-repo)
       "managers": {
         "helmv3": 39,
         "html": 40,
         "pip_requirements": 41,
         "helm-values": 81,
         "github-actions": 108,
         "regex": 933,
         "terraform": 1323
       }
DEBUG: Found github-actions package files (repository=my-tf-repo)
DEBUG: Found helm-values package files (repository=my-tf-repo)
DEBUG: Found pip_requirements package files (repository=my-tf-repo)
DEBUG: Found terraform package files (repository=my-tf-repo)
DEBUG: Found 240 package file(s) (repository=my-tf-repo)
 INFO: Dependency extraction complete (repository=my-tf-repo, baseBranch=master)
       "stats": {
         "managers": {
           "github-actions": {"fileCount": 7, "depCount": 36},
           "helm-values": {"fileCount": 1, "depCount": 1},
           "pip_requirements": {"fileCount": 1, "depCount": 3},
           "terraform": {"fileCount": 231, "depCount": 496}
         },
         "total": {"fileCount": 240, "depCount": 536}
       }

Any hints or ideas of what could be going wrong here would be greatly appreciated!

Logs (if relevant)

Logs

Copy/paste the relevant log(s) here, between the starting and ending backticks

[github-actions[bot]]

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the discussion if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[viceice]

I'm pretty sure your regex manager isn't matching any dependency. It's only matching file.

[Andres-Lu]

Yeah, that's my suspicion as well, but I can't seem to figure out what I'm missing.

https://regex101.com/r/AUCL8m/1

This is the regex string I'm trying to use, is there any escaping i might've missed somewhere when configuring the custom manager?

[Andres-Lu]

I ended up removing the multiline anchors and that did it for me.