caught-error / "Disk space error"

[janpio]

We are super happy users of Renovate, but recently hit a small snag:

What Renovate type, platform and version are you using?

Hosted GitHub App

Describe the bug

One of our repositories https://github.com/prisma/e2e-tests stopped running Renovate successfully.
From https://app.renovatebot.com/ we know that all jobs report caught-error as result.

Relevant debug logs

In the details of a job we can see the following:

...

DEBUG: lock file error(branch="renovate/node-12.x")
{
  "err": {
    "killed": false,
    "code": 1,
    "signal": null,
    "cmd": "docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/prisma/e2e-tests\":\"/mnt/renovate/gh/prisma/e2e-tests\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -e CI -e YARN_HTTP_TIMEOUT -e YARN_ENABLE_SCRIPTS -w \"/mnt/renovate/gh/prisma/e2e-tests/frameworks/nextjs\" docker.io/renovate/node:14.15.4 bash -l -c \"npm i -g yarn && yarn install\"",
    "stdout": "\n> yarn@1.22.10 preinstall /home/ubuntu/.npm-global/lib/node_modules/yarn\n> :; (node ./preinstall.js > /dev/null 2>&1 || true)\n\n/home/ubuntu/.npm-global/bin/yarn -> /home/ubuntu/.npm-global/lib/node_modules/yarn/bin/yarn.js\n/home/ubuntu/.npm-global/bin/yarnpkg -> /home/ubuntu/.npm-global/lib/node_modules/yarn/bin/yarn.js\n+ yarn@1.22.10\nadded 1 package in 1.43s\nyarn install v1.22.10\n[1/4] Resolving packages...\n[2/4] Fetching packages...\ninfo fsevents@2.1.2: The platform \"linux\" is incompatible with this module.\ninfo \"fsevents@2.1.2\" is an optional dependency and failed compatibility check. Excluding it from installation.\ninfo fsevents@1.2.12: The platform \"linux\" is incompatible with this module.\ninfo \"fsevents@1.2.12\" is an optional dependency and failed compatibility check. Excluding it from installation.\n[3/4] Linking dependencies...\ninfo If you think this is a bug, please open a bug report with the information provided in \"/mnt/renovate/gh/prisma/e2e-tests/frameworks/nextjs/yarn-error.log\".\ninfo Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.\n",
    "stderr": "warning ../../package.json: No license field\nerror An unexpected error occurred: \"ENOSPC: no space left on device, copyfile '/home/ubuntu/.cache/yarn/v6/npm-typescript-4.1.3-519d582bd94cba0cf8934c7d8e8467e473f53bb7-integrity/node_modules/typescript/lib/typescriptServices.js' -> '/mnt/renovate/gh/prisma/e2e-tests/frameworks/nextjs/node_modules/typescript/lib/typescriptServices.js'\".\n",
    "message": "Command failed: docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/prisma/e2e-tests\":\"/mnt/renovate/gh/prisma/e2e-tests\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -e CI -e YARN_HTTP_TIMEOUT -e YARN_ENABLE_SCRIPTS -w \"/mnt/renovate/gh/prisma/e2e-tests/frameworks/nextjs\" docker.io/renovate/node:14.15.4 bash -l -c \"npm i -g yarn && yarn install\"\nwarning ../../package.json: No license field\nerror An unexpected error occurred: \"ENOSPC: no space left on device, copyfile '/home/ubuntu/.cache/yarn/v6/npm-typescript-4.1.3-519d582bd94cba0cf8934c7d8e8467e473f53bb7-integrity/node_modules/typescript/lib/typescriptServices.js' -> '/mnt/renovate/gh/prisma/e2e-tests/frameworks/nextjs/node_modules/typescript/lib/typescriptServices.js'\".\n",
    "stack": "Error: Command failed: docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/prisma/e2e-tests\":\"/mnt/renovate/gh/prisma/e2e-tests\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -e CI -e YARN_HTTP_TIMEOUT -e YARN_ENABLE_SCRIPTS -w \"/mnt/renovate/gh/prisma/e2e-tests/frameworks/nextjs\" docker.io/renovate/node:14.15.4 bash -l -c \"npm i -g yarn && yarn install\"\nwarning ../../package.json: No license field\nerror An unexpected error occurred: \"ENOSPC: no space left on device, copyfile '/home/ubuntu/.cache/yarn/v6/npm-typescript-4.1.3-519d582bd94cba0cf8934c7d8e8467e473f53bb7-integrity/node_modules/typescript/lib/typescriptServices.js' -> '/mnt/renovate/gh/prisma/e2e-tests/frameworks/nextjs/node_modules/typescript/lib/typescriptServices.js'\".\n\n    at ChildProcess.exithandler (child_process.js:308:12)\n    at ChildProcess.emit (events.js:314:20)\n    at ChildProcess.EventEmitter.emit (domain.js:483:12)\n    at maybeClose (internal/child_process.js:1021:16)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)"
  },
  "type": "yarn"
}
DEBUG: Passing disk-space error up(branch="renovate/node-12.x")
ERROR: Disk space error - skipping
...

(Taken from https://app.renovatebot.com/dashboard#github/prisma/e2e-tests/293576384)

All our other repositories are working fine.

Is there something we are doing wrong or is this "just" an infrastructure problem?

[HonkingGoose]

If the other repo's don't have this problem, with the same config, and this repo used to work properly but now doesn't, one logical explanation could be that the repo filesize grew too big.

How big is your repository when you do a standard full-fat git clone?
It might be that you're running into file-size restrictions for the Docker container...

There's an issue open to support doing a partial clone for big repositories: https://github.com/renovatebot/renovate/issues/7840

[janpio]

Unfortunately not - but there is a yarn (1) argument to skip these steps (--ignore-scripts) and an env var to disable the code on our side - but I am not sure we can/could set these for the Renovate Cloud app.

[Jolg42]

The failure started 3 hours ago so maybe the termination will help for next run

[rarkins]

In general Renovate does try to ignore scripts.

It's strange you had 3 errors in a row, as they would likely have been on 3 separate machines. Unless all our machines suddenly crossed some threshold necessary to run your repo, it would seem like the failures are related either to changes in your or our source code.

[janpio]

That's also what confuses me: We did not change anything on the repo but update a few dependencies ourselves, and a bunch of Renovate updates as well: https://github.com/prisma/e2e-tests/commits/dev

[Jolg42]

I removed some dependencies, not sure if that did the trick or your restart but it's back to "Done" which means it works.