Grouped updates with git-submodules enabled causes flacky updates

[rarkins]

Discussed in #30187

^{Originally posted by Shegox July 15, 2024}

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

37.431.5

Please tell us more about your question or problem

This issue revisits the problem discussed in #22098, but I am creating a new ticket because the original issue was marked as resolved.

While using this renovate.json configuration and grouping a submodule update with another update:

{
  "extends": ["group:all"],
  "git-submodules": {
    "enabled": true
  }
}

I encounter flaky behavior in updates. Here is a summary of the problematic behavior observed:

1. Renovate detects changes for both the Docker image and my submodule in one run and creates a commit.
2. In the next run, Renovate notes that the Docker image is already up to date on the renovate branch and doesn’t touch it but detects a submodule change, creating a new commit with just the submodule digest.
3. In the subsequent run, Renovate detects that GitHub Actions need an update because the PR only contains the submodule digest bump, prompting a new commit containing all changes.
4. This cycle repeats indefinitely.

The root cause seems to be related to treating git-submodules as special and always marking them as tainted, which forces an update:

renovate/lib/workers/repository/update/branch/get-updated.ts

Lines 276 to 281 in 1c8eb34

	if (newContent === packageFileContent) {
	if (upgrade.manager === 'git-submodules') {
	updatedFileContents[packageFile] = newContent;
	delete nonUpdatedFileContents[packageFile];
	}
	}

Unlike other updates, we don’t rerun with reuseExistingBranch: false for submodules as we do here:

renovate/lib/workers/repository/update/branch/get-updated.ts

Lines 258 to 268 in 1c8eb34

	if (newContent !== packageFileContent) {
	if (reuseExistingBranch) {
	// This ensure it's always 1 commit from the bot
	logger.debug(
	{ packageFile, depName },
	'Need to update package file so will rebase first',
	);
	return getUpdatedPackageFiles({
	...config,
	reuseExistingBranch: false,
	});

Demo

You can see the behavior in this demo repository/update: Demo Pull Request.

Proposed Solutions

I have two potential solutions and would appreciate input on the best approach:

1. Run a Full New Update: Always execute getUpdatedPackageFiles() with reuseExistingBranch: false for git-submodules.
2. Treat Updates Differently: Consider treating submodules (.git index entries specifically) as lockfiles rather than artifacts. This approach, using .isLockfileUpdate, could avoid the need for special handling of package file updates in get-updated.ts.

Looking forward to feedback on these proposed solutions.

Logs (if relevant)

Logs

Update (2 step) from having the right updates to removing the non-submodule update: Shegox/renovate-submodule-group@4d18355

DEBUG: latest commit (repository=Shegox/renovate-submodule-group, branch=renovate/all)
       "branchName": "main",
       "latestCommitDate": "2024-07-15T13:07:19+02:00"
DEBUG: manager.getUpdatedPackageFiles() reuseExistingBranch=true (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: Branch dep node in Dockerfile is already updated (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: No content changed (repository=Shegox/renovate-submodule-group, packageFile=Dockerfile, branch=renovate/all)
       "depName": "node"
DEBUG: updateArtifacts for updatedPackageFiles (repository=Shegox/renovate-submodule-group, branch=renovate/all)
 INFO: Updating submodule renovatebot (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: Updated 1 package files (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: Updated 1 lock files (repository=Shegox/renovate-submodule-group, branch=renovate/all)
       "updatedArtifacts": ["renovatebot"]
DEBUG: Getting comments for #2 (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: http cache: saving https://api.github.com/repos/Shegox/renovate-submodule-group/issues/2/comments?per_page=100 (etag="e23ce81ecd00061fdec06b53eaba189cf18c8c01bb0ec46c10adbd01a50c4aad", lastModified=undefined) (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: Found 0 comments (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: Getting comments for #2 (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: http cache: saving https://api.github.com/repos/Shegox/renovate-submodule-group/issues/2/comments?per_page=100 (etag="e23ce81ecd00061fdec06b53eaba189cf18c8c01bb0ec46c10adbd01a50c4aad", lastModified=undefined) (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: Found 0 comments (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: 2 file(s) to commit (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: Preparing files for committing to branch renovate/all (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: git commit (repository=Shegox/renovate-submodule-group, branch=renovate/all)
       "deletedFiles": [],
       "ignoredFiles": [],
       "result": {
         "author": null,
         "branch": "renovate/all",
         "commit": "4d18355dd7276c1391c729f3f10f77e15a58db30",
         "root": false,
         "summary": {"changes": 1, "insertions": 1, "deletions": 1}
       }
DEBUG: Pushing refSpec renovate/all:renovate/all (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: git push (repository=Shegox/renovate-submodule-group, branch=renovate/all)
       "result": {
         "pushed": [],
         "ref": {"local": "refs/remotes/origin/renovate/all"},
         "remoteMessages": {"all": []}
       }
DEBUG: Setting current branch to main (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: latest commit (repository=Shegox/renovate-submodule-group, branch=renovate/all)
       "branchName": "main",
       "latestCommitDate": "2024-07-15T13:07:19+02:00"
 INFO: Branch updated (repository=Shegox/renovate-submodule-group, branch=renovate/all)
       "commitSha": "4d18355dd7276c1391c729f3f10f77e15a58db30"
DEBUG: Ensuring PR (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: There are 0 errors and 0 warnings (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: getBranchPr(renovate/all) (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: findPr(renovate/all, undefined, open) (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: Found PR #2 (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: getPrCache() (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: Found existing PR (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: Fetching changelog: https://github.com/nodejs/node (18.0.0 -> 20.15.1) (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: Processing existing PR (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: setPrCache() (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: Pull Request #2 does not need updating (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: PR is not configured for automerge (repository=Shegox/renovate-submodule-group, branch=renovate/all)
DEBUG: setBranchCommit() (repository=Shegox/renovate-submodule-group, branch=renovate/all)

[rarkins]

Reproduction forked to https://github.com/renovate-reproductions/30187