Massive performance issue on very large repo

[HenrikPoulsen]

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

34.159.1

If you're self-hosting Renovate, select which platform you are using.

GitHub Enterprise Server

If you're self-hosting Renovate, tell us what version of the platform you run.

3.7.6

Was this something which used to work for you, and then stopped?

I never saw this working

Wanted end result.

Renovate intermittently takes a very long time to run on a large monorepo. 15k branches and many hundred active contributors.
The thing that stands out in the logs is what you see in the debug logs section.
Took a look in the code and that seems to run git branch --remotes --verbose --contains {sha}

Running that on my computer with a time prepended to it gives the following:

real    11m9.551s
user    11m8.929s
sys     0m0.110s

So it is extremely slow on that repo.
Is there any other way that code can accomplish what it needs without enumerating all 15k branches?
What are the options available to me to prevent this code to run at all? Is there such a method?

What you tried so far.

Haven't tried anything other than that I was able to reproduce that it is in fact a very slow command

Relevant debug logs

Logs

2023-03-09T09:51:32.3977423Z DEBUG: branch.isBehindBase(): using git to calculate (repository=org/repo)
2023-03-09T10:09:58.1194894Z DEBUG: branch.isBehindBase(): false (repository=org/repo)

[rarkins]

That particular git command is part of the "is the branch behind base branch?" logic:

renovate/lib/util/git/index.ts

Lines 567 to 606 in a3cc159

	export async function isBranchBehindBase(
	branchName: string,
	baseBranch: string
	): Promise<boolean> {
	let isBehind = getCachedBehindBaseResult(
	branchName,
	getBranchCommit(branchName), // branch sha
	baseBranch,
	getBranchCommit(baseBranch) // base branch sha
	);
	if (isBehind !== null) {
	logger.debug(`branch.isBehindBase(): using cached result "${isBehind}"`);
	return isBehind;
	}
	logger.debug('branch.isBehindBase(): using git to calculate');
	await syncGit();
	try {
	const { currentBranchSha, currentBranch } = config;
	const branches = await git.branch([
	'--remotes',
	'--verbose',
	'--contains',
	config.currentBranchSha,
	]);
	isBehind = !branches.all.map(localName).includes(branchName);
	logger.debug(
	{ currentBranch, currentBranchSha },
	`branch.isBehindBase(): ${isBehind}`
	);
	return isBehind;
	} catch (err) /* istanbul ignore next */ {
	const errChecked = checkForPlatformFailure(err);
	if (errChecked) {
	throw errChecked;
	}
	throw err;
	}
	}

It looks like we're not caching the result, but even doing so would not be too helpful if it takes 11minutes to get a response.

[rarkins]

Can you check out the branch from #20837 and confirm if that also solves you problem?

[HenrikPoulsen]

Haven't tried it yet. Don't see how it would work though tbh. setCachedBehindBaseResult() isn't called anywhere in the code, as mentioned in the PR.
But in my case I don't want it to be called at all. Since even running it once per PR is still way too much. Is the intent to cache undefined when the check is to be skipped?

[rarkins]

Yes

[HenrikPoulsen]

I can confirm that it does not call the slow command with the branch you mentioned, it sets isBehindBase to undefined and then moves on.
So from that perspective it seems good.
Does the cached value get set somewhere else? Since as I mentioned setCachedBehindBaseResult() isn't called anywhere. Without knowing that part of the code it just seems like that is what you would expect. Set is called to cache it, then get to fetch it later. Or is somewhere in the code inserting that directly in the cache without calling that function? If so then I imagine it's fine to ship 🤞

[rarkins]

Will fix that first

[rarkins]

Related: #20837 and #20839

[kwelch]

Note: not an answer.

I am just curious why Renovate is using git branch --remotes --verbose --contains {sha} this command specifically to determine if the branch is behind. Why are we checking all branches? Shouldn't this just be a check between the update branch and base branch?

I have not have examined this deeply, but we are seeing this error resulting in hours of processing, so I am trying to understand it better so we can help bring that time down.

[rarkins]

This is an easy way to know whether a branch is derived from another commit. Even ChatGPT recommended this approach when I asked it today:

[HonkingGoose]

Here's the official git branch manual: ¹

With --contains, shows only the branches that contain the named commit (in other words, the branches whose tip commits are descendants of the named commit), --no-contains inverts it.

I also found this summary in the git branch manual: ²

The options --contains, --no-contains, --merged and --no-merged serve four related but different purposes:

• --contains <commit> is used to find all branches which will need special attention if <commit> were to be rebased or amended, since those branches contain the specified <commit>.
• --no-contains <commit> is the inverse of that, i.e. branches that don’t contain the specified <commit>.
• --merged is used to find all branches which can be safely deleted, since those branches are fully contained by HEAD.
• --no-merged is used to find branches which are candidates for merging into HEAD, since those branches are not fully contained by HEAD.

Footnotes

1. https://git-scm.com/docs/git-branch#_description ↩

2. https://git-scm.com/docs/git-branch#_notes ↩

[rarkins]

I have rewritten this function in #21036 to not use the origin and will merge it after more testing

[kwelch]

OMG! Huge thanks! I likely need to run more tests since these numbers don't feel accurate, but I did a dry-run against our longest running repository using this patch and our current live version. This new version finished in 47mins, and well the other has been running for 17 hours.

[rarkins]

Happy to hear more once you're done testing. I was surprised to not see any performance boost in the hosted app after deploying, in fact performance went down a bit at the time but that can be due to a number of things