Renovate intermittently failing because of "SIGSEGV" signal

[amimas]

What would you like help with?

Other

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

{'platform': 'gitlab', 'version': '~37'}

Please tell us more about your question or problem

Hi,

I'm self-hosting renovate and recently it started to fail intermittently. When it fails, I have the following error in the terminal output but no relevant info about this in the log.

error Command failed with signal "SIGSEGV".

The way I run renovate is following:

• Installing renovate using yarn with following package.json and relevant yarn.lock file.

{
  "name": "renovate",
  "private": true,
  "version": "1.0.0",
  "type": "commonjs",
  "scripts": {
    "renovate": "renovate",
    "validate-config": "renovate-config-validator"
  },
  "dependencies": {
    "dotenv": "16.4.5",
    "renovate": "~37"
  },
  "engines": {
    "node": "~20",
    "yarn": "^1.22.0",
    "npm": "please use yarn"
  }
}

• In GitLab pipeline I setup docker-in-docker and then install renovate. After this, run renovate using yarn renovate (from package.json above). Below is my GitLab CI config for running renovate.

run-renovate:
  tags:
    - docker:privileged
  image: docker:25.0.0-dind-alpine3.19
  services:
    - name: docker:25.0.0-dind-alpine3.19
      alias: docker
  variables:
    DOCKER_AUTH_CONFIG: |
      {"auths":{"quay.io":{"auth":"${****}"}}}
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
    DOCKER_HOST: tcp://docker:2375
    RENOVATE_CLI_FLAGS: ""
    ## Variables for configuring renovate bot's access to resources
    GITLAB_API_URL: ${CI_API_V4_URL}
    RENOVATE_BASE_DIR: ${CI_PROJECT_DIR}/renovate
    RENOVATE_TOKEN: ${GITLAB_USER_TOKEN_WRITEREPO}
    GITHUB_COM_TOKEN: ${GITHUB_USER_TOKEN_READREPO}
    RENOVATE_GIT_AUTHOR: GitLab Service Bot <gitlab-service-bot1@gitlab.example.com>
    DOCKERHUB_AUTH_USERNAME: ${DOCKERHUB_USERNAME}
    DOCKERHUB_AUTH_PASSWORD: ${DOCKERHUB_ACCESS_TOKEN}
    QUAY_AUTH_USERNAME: ${QUAY_USERNAME_****}
    QUAY_AUTH_PASSWORD: ${QUAY_PASSWORD_****}
    QUAY_OAUTH_TOKEN: ${QUAY_OAUTH_TOKEN_****_OAUTHAPP}
  before_script:
    # Prepare renovate directory
    - mkdir ${RENOVATE_BASE_DIR}
    # Install project depedencies and tools
    - apk add --no-cache git=~2 nodejs=~20 npm=~10 yarn=~1
    - yarn install
    # Test docker is working
    - docker --version
    - docker ps --all
  script:
    - yarn renovate ${RENOVATE_CLI_FLAGS}
 

As you can see, I'm running renovate in the GitLab pipeline using node v20. The process runs on AWS EC2 instance type t3.medium, which has 2 CPU and 4GB memory.

This has been working fine for several years. As of 2 weeks ago, renovate started to fail intermittently with the signal error I mentioned above. I had enabled renovate to keep itself up-to-date. Going through all the pipelines where it failed, I think it started failing when renovate was updated from 37.408.3 -> 37.414.1.

Going through the changelog of that version range, I'm not sure what (or if any) of them are the root cause. I also can't find any relevant info in the log file. It just abruptly ends wherever renovate process failed.

Not sure how to provide a reproducible public repo/execution. Could you please help or advise what I can do next?

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks

[viceice]

use our gitlab pipeline template to run renovate on gitlab runner

https://gitlab.com/renovate-bot/renovate-runner

[viceice]

if you run our docker image directly, then you don't need dind at all, which will also speed up your runs

[amimas]

Well... Reason for dind is because the CI environment is also running in docker. My understanding is that renovate will pull other helper docker image for required package manager to be able to perform required update. Not sure how using the official renovate docker image helps with that. I haven't looked into the official image or template deeply.

Anyways, I still have doubts that this has anything to do with the CI environment. It's been running fine for years and we haven't made any changes to our CI environment when the issue started. Only relevant thought I have is that perhaps the amount of memory available to renovate is not enough anymore after the version update I mentioned. But, in that case I would've expected out-of-memory type errors but that's not what we're getting.

[viceice]

please try our default docker image, it doesn't need docker anymore. it installs tool on the fly and ensures a compatible system environment.

this is my last answer, if no more time for personal support.

[amimas]

Hi, Writing back to give an update on this.

I made changes to how renovate was being executed. Instead of executing it via npm package of renovate in a docker-in-docker environment, switched to official renovate image in GitLab CI. Also, upgraded to latest major version 38. Combining both of these setups, now I don't see the issue reported originally.

In my opinion, it still doesn't explain the root cause of the issue I reported, but I'm happy to have a simpler setup that doesn't involve docker-in-docker.

Thanks for your help.

[viceice]

Hi, Writing back to give an update on this.

I made changes to how renovate was being executed. Instead of executing it via npm package of renovate in a docker-in-docker environment, switched to official renovate image in GitLab CI. Also, upgraded to latest major version 38. Combining both of these setups, now I don't see the issue reported originally.

In my opinion, it still doesn't explain the root cause of the issue I reported, but I'm happy to have a simpler setup that doesn't involve docker-in-docker.

Thanks for your help.

it's probably something with the node version or some other system package update there which caused it 🤷‍♂️