lookupUpdates error for Gemfile

[mvitz]

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

Self-hosted GitLab, Renovate v37.415.0

Please tell us more about your question or problem

Since Friday, our internal renovate job fails most of the time with a lookupUpdates error (see Logs).

I'm unfortunately unable to pin the error to a specific repository or dependency because these change from time to time. And I already suspected and tested en-/disabling caching between multiple runs, but this did not fix it either.

If more configuration settings are required for testing this, I'm happy to provide these.

Sorry that I'm unable to provide a reproducer or any more information. Hope somebody can help me.

Logs (if relevant)

Logs

...
ERROR: lookupUpdates error (repository=XXX, packageFile=Gemfile)
       "currentDigest": undefined,
       "currentValue": "'~> 9.4'",
       "datasource": "rubygems",
       "packageName": "octicons_helper",
       "digestOneAndOnly": undefined,
       "followTag": null,
       "lockedVersion": "9.6.0",
       "pinDigests": false,
       "rollbackPrs": false,
       "isVulnerabilityAlert": undefined,
       "updatePinnedDependencies": true,
       "unconstrainedValue": false,
       "err": {
         "message": "Cannot read properties of null (reading 'compare')",
         "stack": "TypeError: Cannot read properties of null (reading 'compare')\n    at gt (/usr/local/renovate/node_modules/.pnpm/@renovatebot+ruby-semver@3.0.23/node_modules/@renovatebot/ruby-semver/dist/index.js:21:14)\n    at Object.isGreaterThan (/usr/local/renovate/lib/modules/versioning/ruby/index.ts:52:5)\n    at /usr/local/renovate/lib/workers/repository/process/lookup/filter.ts:41:18\n    at Array.filter (<anonymous>)\n    at filterVersions (/usr/local/renovate/lib/workers/repository/process/lookup/filter.ts:38:35)\n    at lookupUpdates (/usr/local/renovate/lib/workers/repository/process/lookup/index.ts:350:44)\n    at Function.wrap (/usr/local/renovate/lib/util/stats.ts:38:20)\n    at fetchDepUpdates (/usr/local/renovate/lib/workers/repository/process/fetch.ts:58:42)\n    at /usr/local/renovate/lib/workers/repository/process/fetch.ts:104:23\n    at /usr/local/renovate/node_modules/.pnpm/p-map@4.0.0/node_modules/p-map/index.js:57:22"
       }
...
DEBUG: Rubygems: error fetching rubygems data, falling back to versions-only result (repository=XXX)
       "err": {"type": "cache-invalid"}
...

[rarkins]

It seems like theres a bug in the underlying Ruby semver library but it's very hard to know how to fix it without having a reproduction or knowing which version triggered it.

[mvitz]

Is there anything I can do to provide additional information? I tried running it with TRACE logging enabled, but because we have many repositories, this produces a multiple GB log file.
Maybe preserving the cache to look into a file there or something?

[rarkins]

Work out which dependency causes it, then try to reproduce it in a test repo

[Goltergaul]

I'm seeing the same issue with our repositories. I'll try if i can find some additional infos

[ChristianCiach]

I updated from 38.25.1 to 38.40.0 and this seems to have fixed it.

[mvitz]

For us, the first three consecutive runs all succeeded, but just a few seconds ago the fourth run failed with the lookupUpdates error again.

[BjoernPetersen]

We're observing the same error (currently on Renovate v38.39.6) for projects with a Gemfile that's literally just this:

source "https://rubygems.org"

gem "fastlane"

Corresponding Gemfile.lock (had to add a .txt ending so GitHub accepts the file)

The error only appears sometimes, so it's hard to reproduce or pin down the root cause. The bug was introduced somewhere between Renovate v37.89.7 and v38.20.1 (we jumped between those versions due to a template change).

[ChristianCiach]

I am sorry, I looked again. There is actually one other Gemfile hidden deep inside another repository that is scanned by Renovate (but in the same Gitlab group, so there is a chance that both repos are processed shortly after another). So please don't rule out your idea just yet :)

[ChristianCiach]

I cannot reproduce this locally. Do you guys use some kind of caching? We use both repositoryCache (which was once "experimental", but seems to be stable now) and redisUrl.

[BjoernPetersen]

We're using this exact template: https://gitlab.com/renovate-bot/renovate-runner, so we're using repository caching.

[mvitz]

I wasn't able to reproduce this locally once. The only thing I noticed was that, when running within the Docker Image, I wasn't able to execute renovate more than once because of some circumstances the Node.js version changed during the first run and wasn't able to start renovate after that.

When trying to reproduce this, the first thing I did was to disable the repositoryCache. But this didn't help. Afterwards, I tried to reproduce the error with a single repository, which didn't work either. Thereafter, I added one repository more. This didn't reproduce the error, either. I repeated that step and after some number of repositories the error occurred once again. I then attempted to only ignore that repo, but it didn't help either.

Since then, I gave up because I've no clue how to debug this any further.

[viceice]

I wasn't able to reproduce this locally once. The only thing I noticed was that, when running within the Docker Image, I wasn't able to execute renovate more than once because of some circumstances the Node.js version changed during the first run and wasn't able to start renovate after that.

which version are you using? the latest docker images are using a separate node version to run renovate, so multiple runs should be possible.

When trying to reproduce this, the first thing I did was to disable the repositoryCache. But this didn't help. Afterwards, I tried to reproduce the error with a single repository, which didn't work either. Thereafter, I added one repository more. This didn't reproduce the error, either. I repeated that step and after some number of repositories the error occurred once again. I then attempted to only ignore that repo, but it didn't help either.

Since then, I gave up because I've no clue how to debug this any further.

[rybnico]

We also ran into this bug and this is what I found after some debugging/testing.
I nailed it down to the vtrim function - sometimes the string returned by this function doesn't contain the expected (trimmed) string, but some gibberish instead. I was able to catch the bug in the debugger. What I did was to compare the version returned by the cached regEx with an uncached one.

As you can see in the attached screenshot, the version.replace for newVersion returns some invalid characters/bytes, while the version.replace with the uncached regEx for newVersionUncached returns the expected string.

The problematic code can be found here:

renovate/lib/util/regex.ts

Line 49 in bf2f3c3

const cachedResult = cache.get(key);

The cached regEx seems to be corrupted somehow.
Any idea why?

[ChristianCiach]

Since we cannot reproduce it, I wonder if maybe one of these URLs just return some corrupted contents sometimes:

DEBUG: http statistics (repository=local)
       "urls": {
         "https://rubygems.org/api/v1/gems/activesupport.json (GET,200)": 1,
         "https://rubygems.org/api/v1/gems/sinatra-cross_origin.json (GET,200)": 1,
         "https://rubygems.org/api/v1/gems/sinatra.json (GET,200)": 1,
         "https://rubygems.org/api/v1/versions/activesupport.json (GET,200)": 1,
         "https://rubygems.org/api/v1/versions/sinatra-cross_origin.json (GET,200)": 1,
         "https://rubygems.org/api/v1/versions/sinatra.json (GET,200)": 1,
         "https://rubygems.org/versions (GET,200)": 1
       },

I've queried these URLs many times and didn't see anything strange, but I am really out of ideas.

EDIT Sorry, please disregard that. I now see that in your example, version is still fine but newVersion gets corrupted by the regex replaces. This surely has nothing to do with the API responses.

[viceice]

we need to create a test case with normal regex and re2, so we can see if it's a cache bug or a re2 bug

[rybnico]

I had a go at creating a test case for this, but had no success. I suspect it has something to do with garbage collection/memory management.

[viceice]

I think this is now fixed

• Non-deterministic matching starting in v1.21.2 uhop/node-re2#214
• build(deps): update dependency re2 to v1.21.4 #31131

can you please verify?

[mvitz]

At least since updating to 38.58.2 which contains the mentioned fix, none of our pipelines failed with a lookupUpdates error. Therefore, I would assume that this is indeed fixed by the re2 update.