Recursive git submodules

[Josef-Reichardt]

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitLab (ghcr.io/renovatebot/renovate:37.429.0)

Please tell us more about your question or problem

We have a rust repo which has dependency on a crate which is located in a sub-module of a sub-module. But the problem is not rust specific.

Example directory structure:

• main repo
  • demo-app (with dependency on demo-lib
  • sub1 (a git submodule containing again a git submodule)
    • sub2 (a git submodule containing demo-lib)

You can clone the whole repo tree with git clone --recursive ... or update the submodules with git submodule update --recursive.

We have already set the toggle --clone-submodules and this works for the renovate bot when he works directly on the repo sub1 in this example. But it doesn't work for the main repo because the bot only loads one level ov submodules but he don't do it recursively.

I have created a reproduction repo on GitHub where you can see the same behaviour: https://github.com/Josef-Reichardt/renovate-bot-reproduction-recursive-submodules/
And here you can find a pull request from the renovate bot where he shows an error because of the missing sub-submodule: Josef-Reichardt/renovate-bot-reproduction-recursive-submodules#3

Logs (if relevant)

Logs

Command failed: cargo update --config net.git-fetch-with-cli=true --manifest-path demo-app/Cargo.toml --workspace
error: failed to get `renovate-bot-reproduction-recursive-submodules-sub2` as a dependency of package `renovate-bot-reproduction-recursive-submodules v0.1.0 (/tmp/renovate/repos/github/Josef-Reichardt/renovate-bot-reproduction-recursive-submodules/demo-app)`

Caused by:
  failed to load source for dependency `renovate-bot-reproduction-recursive-submodules-sub2`

Caused by:
  Unable to update /tmp/renovate/repos/github/Josef-Reichardt/renovate-bot-reproduction-recursive-submodules/renovate-bot-reproduction-recursive-submodules-sub1/renovate-bot-reproduction-recursive-submodules-sub2

Caused by:
  failed to read `/tmp/renovate/repos/github/Josef-Reichardt/renovate-bot-reproduction-recursive-submodules/renovate-bot-reproduction-recursive-submodules-sub1/renovate-bot-reproduction-recursive-submodules-sub2/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

[rarkins]

Renovate doesn't do a regular clone for performance optimization reasons. When doing a branch checkout it does attempt to recurse submodules though:

renovate/lib/util/git/index.ts

Lines 530 to 534 in 17eacd2

	git.checkout(
	submodulesInitizialized
	? ['-f', '--recurse-submodules', branchName, '--']
	: ['-f', branchName, '--'],
	),

I tried adding --recurse to this logic:

renovate/lib/util/git/index.ts

Lines 354 to 364 in 17eacd2

	for (const submodule of submodules) {
	try {
	logger.debug(`Cloning git submodule at ${submodule}`);
	await gitRetry(() => git.submoduleUpdate(['--init', submodule]));
	} catch (err) {
	logger.warn(
	{ err },
	`Unable to initialise git submodule at ${submodule}`,
	);
	}
	}

But it didn't fix your reproduction repo.

I've converted this to an issue as I don't have time to look into it any further at the moment: #30294

If you are able to find the solution, a PR would be welcome

[Josef-Reichardt]

Thanks for your response. I've privided a PR: #30361