Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GitLab: Two pipelines per MR #5743

Closed
CommonGuy opened this issue Mar 18, 2020 · 11 comments
Closed

GitLab: Two pipelines per MR #5743

CommonGuy opened this issue Mar 18, 2020 · 11 comments
Labels
platform:gitlab GitLab Platform

Comments

@CommonGuy
Copy link

CommonGuy commented Mar 18, 2020 •
edited
Loading

What Renovate type are you using?

We are using the Docker version (renovate/renovate:19) on https://gitlab.com. It is scheduled to run all 2 hours.

Describe the bug

We configured renovate with stabilityDays: 3. In GitLab, this creates a new job, which makes the MR wait for the specified time. So far, all good.

However, sporadically the renovate/stability-days job will be placed on a new pipeline instead of the existing pipeline. Note that two pipelines exist instead of one. The newer pipeline with only one job is the renovate/stability-days job.
image

Usually, the renovate/stability-days job will be placed correctly in the existing pipeline (the job on the far right in this screenshot):
image

Creating two pipelines has a huge impact: In the merge request, only the most recent pipeline is checked. So if our testing-pipeline fails, the merge request can still be merged (because the renovate/stability-days job passes).

Did you see anything helpful in debug logs?

Debug-Logs of a case:

DEBUG: generateBranchConfig(1) (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: hasGroupName: false (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: groupEligible: false (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: useGroupSettings: false (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: generateBranchConfig(1) (repository=(redacted), branch=renovate/major-workbox-monorepo)
DEBUG: hasGroupName: true (repository=(redacted), branch=renovate/major-workbox-monorepo)
DEBUG: groupEligible: false (repository=(redacted), branch=renovate/major-workbox-monorepo)
DEBUG: useGroupSettings: false (repository=(redacted), branch=renovate/major-workbox-monorepo)
DEBUG: config.repoIsOnboarded=true (repository=(redacted))
DEBUG: Processing 13 branches: (redacted...) renovate/webpack-bundle-analyzer-3.x (repository=(redacted))
DEBUG: Calculating hourly PRs remaining (repository=(redacted))
DEBUG: currentHourStart=1584543600000 (repository=(redacted))
DEBUG: PR hourly limit remaining: 2 (repository=(redacted))
DEBUG: Enforcing prConcurrentLimit (20) (repository=(redacted))
DEBUG: 2 PRs are currently open (repository=(redacted))
DEBUG: PR concurrent limit remaining: 18 (repository=(redacted))
DEBUG: processBranch with 1 upgrades (repository=(redacted), dependencies=webpack-bundle-analyzer, branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Setting baseBranch to master (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Setting baseBranch to master (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: latest commit (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
       "branchName": "master",
       "latestCommitDate": "2020-03-18 14:57:57 +0000"
DEBUG: getBranchPr(renovate/webpack-bundle-analyzer-3.x) (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Got res with 1 results (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: getPr(145) (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: pr cannot be merged (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: branchExists=true (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Branch pr rebase requested: false (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Branch has 1 upgrade(s) (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Checking if PR has been edited (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Found existing branch PR (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Checking schedule(at any time, null) (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: No schedule defined (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Branch already exists (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: getBranchPr(renovate/webpack-bundle-analyzer-3.x) (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Got res with 1 results (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: getPr(145) (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: pr cannot be merged (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Branch is stale and needs rebasing (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Using parentBranch: undefined (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: manager.getUpdatedPackageFiles() (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: npm.updateDependency(): devDependencies.webpack-bundle-analyzer = 3.6.1 (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Updating packageFile content (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Updated 1 package files (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Getting updated lock files (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: lock file dirs (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
       "dirs": {
         "yarnLockDirs": [],
         "npmLockDirs": ["package-lock.json"],
         "pnpmShrinkwrapDirs": [],
         "lernaDirs": []
       }
DEBUG: Writing package.json files (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
       "packageFiles": ["package.json"]
DEBUG: Writing package-lock.json (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Writing any updated package files (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Writing package.json (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Generating package-lock.json for . (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Spawning npm install to create /tmp/renovate/repos/gitlab/(redacted)/package-lock.json (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: No locally installed npm found (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Could not find globally nested npm (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Using npm: node /usr/lib/node_modules/npm/bin/npm-cli.js install --package-lock-only --no-audit (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Executing command (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
       "command": "node /usr/lib/node_modules/npm/bin/npm-cli.js install --package-lock-only --no-audit"
DEBUG: exec completed (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
       "cmd": "node /usr/lib/node_modules/npm/bin/npm-cli.js install --package-lock-only --no-audit",
       "seconds": 11,
       "stdout": "added 2083 packages in 9.826s\n",
       "stderr": "npm WARN @(redacted)@0.0.0-development No repository field.\n\n"
DEBUG: package-lock.json needs updating (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Updated 1 lock files (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
       "updatedArtifacts": ["package-lock.json"]
DEBUG: 2 file(s) to commit (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Committing files to branch renovate/webpack-bundle-analyzer-3.x (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
 INFO: Branch updated (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
       "commitHash": "261dbde"
DEBUG: Got res with 0 results (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Updating renovate/stability-days status check state to green (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Checking if we can automerge branch (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: mergeStatus=no automerge (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Ensuring PR (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: There are 0 errors and 0 warnings (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: getBranchPr(renovate/webpack-bundle-analyzer-3.x) (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Got res with 1 results (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: getPr(145) (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: getBranchStatus(renovate/webpack-bundle-analyzer-3.x) (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Got res with 9 results (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Found existing PR (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: getBranchStatus(renovate/webpack-bundle-analyzer-3.x) (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Got res with 9 results (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Processing existing PR (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: getBranchStatus(renovate/webpack-bundle-analyzer-3.x) (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Got res with 9 results (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: getBranchStatus() result (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
       "branchStatus": "yellow",
       "branchName": "renovate/webpack-bundle-analyzer-3.x"
DEBUG: Merge Request #145 does not need updating (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Ensuring comment ":warning: Artifact update problem" in #145 is removed (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Getting comments for #145 (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Found 3 comments (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: Checking #145 for automerge (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
       "automerge": true,
       "automergeType": "pr",
       "automergeComment": "automergeComment"
DEBUG: PR is configured for automerge (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
DEBUG: PR is not ready for merge (repository=(redacted), branch=renovate/webpack-bundle-analyzer-3.x)
       "canMergeReason": undefined

Note: this debug case is special, the renovate/stability-days job was created before the actual jobs:
image

To Reproduce

Since this only happens sporadically, it is hard to exactly reproduce this problem. Maybe it is a problem of GitLab...

Additional context

The problem happens for some merge request, while others work correctly in the same repository. Of course, disabling stabilityDays would prevent this problem, since the additional job wouldn't be created.
@rarkins
Copy link
Collaborator

rarkins commented Mar 18, 2020

Keeping debug logs would help with this, especially for the job that creates the extra/incorrect pipeline. Also, once such an MR exists it would be very good to query the API manually to get a complete list of its checks. I definitely don't rule out a Renovate bug here.

@CommonGuy
Copy link
Author

CommonGuy commented Mar 18, 2020 •
edited
Loading

@rarkins I added the debug logs (is it enough?). Note that the case I observed was even more strange. It created a single pipeline for the stability-days job, then created the normal pipelines.

@viceice
Copy link
Member

viceice commented Mar 18, 2020

i think this is a gitlab bug, because renovate puts the status check on commits, not on pipelines.

Maybe something is triggering two pipelines on a pr update?

@rarkins rarkins closed this as completed Apr 7, 2020
@bytestream
Copy link
Contributor

I have the exact same issue. Debug logs are permanently enabled but they don't seem to log anything regarding pipelines for stability days.

@bytestream
Copy link
Contributor

I may be something to do with a recent GitLab upgrade as for a while renovate/stability-days was a job appended to the default pipeline. Recently it's become a separate pipeline which seems to cause some kind of race condition, sometimes renovate/stability-days is generated after the repository pipeline, sometimes it's generated before.

@bytestream
Copy link
Contributor

https://gitlab.com/gitlab-org/gitlab/-/issues/14064

I think this is a renovate bug?

@bytestream
Copy link
Contributor

https://docs.gitlab.com/ee/api/commits.html#post-the-build-status-to-a-commit

On this line https://github.com/renovatebot/renovate/blob/master/lib/platform/gitlab/index.ts#L655, we need to pass ref with branchSha

@bytestream bytestream mentioned this issue May 27, 2020
Closed
@andrewgies17
Copy link

I'm still able to reproduce this issue on GitLab.com. If it's a race condition (as theorized in the linked PR before it was closed), can renovate wait for some condition to be true before attempting to create the status check?

As it stands, this makes auto-merge entirely useless because there's a nonzero chance that code will be merged without passing our actual pipeline, just because the update has existed for X days. This also means that the stability-days feature doesn't work as intended - if the actual pipeline is created after the stability-days pipeline, the stability-days pipeline will have no effect on whether the MR can be merged.

The original gitlab issue that this was blamed on was closed/fixed nearly a year ago, so this is presumably either a renovate issue or a different gitlab issue.

@viceice
Copy link
Member

viceice commented Sep 3, 2020

@andrewgies17 please provide a public reproduction repo. As I noted above, this seems to be a gitlab bug.

@bytestream
Copy link
Contributor

I was unable to replicate on gitlab.com so decided to remove stability days to prevent the issue and avoid wasting any further time on it. If you have a public reproducer please share it @andrewgies17 !

@geraintwhite
Copy link
Contributor

@andrewgies17 in my instance I use an external CI pipeline (Jenkins) which reports to GitLab. The Jenkins jobs sometimes take a while to start and if the stability days status check gets applied first, the MR can be merged before the Jenkins job has started. It would be useful if there was an option to only apply the stability days status check if there is already a passing pipeline.

@rarkins rarkins mentioned this issue Nov 24, 2020
Closed
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 25, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
platform:gitlab GitLab Platform
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(gitlab): Include ref in /projects/:id/statuses/:sha call bytestream/renovate
6 participants
@bytestream @viceice @geraintwhite @rarkins @CommonGuy @andrewgies17