Build triggered needlessly after MR states change

[1100101]

Issue

Our gitlab is set up so that each merge request must be approved by 1 or 2 reviewers before it can be merged. In addition Jenkins has to successfully build the project with the proposed changes.
When a reviewer hits the "Approve" button, the "Merge Request" webhook is triggerd, causing Jenkins to build the project again, even if it was successfully built previously.

Context

• Gitlab plugin version: 1.4.3
• Gitlab version: 9.2.10-ee
• Jenkins version: 2.46.2
• Job type: Freestyle

Logs & Traces

Problem description

Gitlab sends a "Merge Request" json with action "approved".
That action is apparently not evaluated, and it is also not checked whether there were actual changes committed to the feature branch.

Expected:
Hitting "Approve" in Gitlab does not trigger a new build in Jenkins.

[bzon]

Do you have Push Events checked?

To accomplish what you expect, you should just have Merge Request events checked and ensure Push Events is not checked. The workaround here is that you have to enforce that developers should have a Merge Request every after feature branch creation and use [WIP] if they don't want to run Jenkins build every push.

[1100101]

Nope, Push Events is not checked, only Merge Request and Comment (the last one is so Jenkins rebuilds on a specific comment).

[bzon]

How many webhooks do you have in the repository? In the job that this webhook triggers, is the Push Events unchecked as well?

[1100101]

The project has 4 web hooks configured, each talking to a different Jenkins job (all Freestyle jobs).
The Jenkins job in question only has Gitlab Triggers "Merge Request" and "Comment" enabled.

Let me know if you need more information

[bzon]

Ok then that is weird, we basically have the same configuration. I'm using gitlab 9.4.3-ce.

[1100101]

But the CE version does not have the "Approvers" functionality, right?
I see in the log that Gitlab triggers the "Merge Request" hook (for either each approval, or just once when all required approvals are available, I'm not sure).
The incoming JSON from Gitlab mentions "action: approved"

The question is now: Is it a Gitlab issue (Maybe even fixed in a newer version), or is it a feature?
I don't know :/

[trunov-ms]

What's the size of build history of this Jenkins job?

[1100101]

The job retains 30 days of build records

[fuzza]

Hi guys,

It seems that we have exactly the same issue as @1100101 mentioned.

The issue itself is that when approver hits Approve button on Gitlab it submits a Merge Request event hook with action: approved in payload and it triggers a new Jenkins job.

It's unexpected behavior because there no code changes is made since previous build that was triggered when merge request is opened or updated (by pushing new changes). It also seems to be a relatively new behavior because the same job setup was working correctly a while ago.

I'm not a Java developer, but from what I see in code, there are only two possible cases for Action enum (open, update) while currently there are three possible actions that we receive in action field of Merge Request hooks - open, updated, approved.

What do you think, could such issue be caused by some changes in Gitlab webhooks API or some regression in Gitlab plugin?

Thanks in advance!

Context

Gitlab plugin version: 1.4.8
Gitlab version: 9.5.5-ee cfe2d5c
Jenkins version: 2.71.3
Job type: Pipeline

Gitlab web hooks
Push events
Merge request events
Comments

Gitlab plugin settings

• - Push Events
• - Opened Merge Request Events
• - Accepted Merge Request Events  
• - Closed Merge Request Events 
• - Comments

Rebuild open Merge Requests - On push to source branch
Comment for triggering a build - /retest

[tyrone-grech]

Hi all,

We are having a similar issue. We are using GitLab Plugin version 1.4.8 and GitLab Server version 10.0.2-ee.

With Opened Merge Request Events ticked in Jenkins configuration, every time we update a Merge Request at all (e.g. add or remove an Approver, change Assignee, etc.) a new job is being triggered. We have temporarily toggled off Opened Merge Request Events and are using Push Events and Comments instead, but what we would like to have is the ability to trigger a job on a subset of Merge Request Events that can actually break builds (e.g. changing target branch).

[1100101]

I think I have an idea why this might be happening to me:
The git repository that triggers the Jenkins job is actually not the one that is checked out by the job.
Due to the fact that we use a custom build framework/toolchain, the workspace is actually consumed by some other repository.
We use scripts to then manually switch the (sub-)module that triggered the job to the feature branch.

So my though is this: In MergeRequestHookTriggerHandlerImpl.isLastCommitNotYetBuild(), it compares the SHA1 from the Merge Request webhook JSON response (commit SHA1 from the MR module), to the SHA1 commits of the repository actually checked out by Jenkins.
This would of course never match, and therefore would trigger a new build even though there was no code change.

Am I correct in this?

[1100101]

Can anyone provide some feedback on this?
To the commenters who experience the same issue: Is your setup similar to ours, as described in my last comment?
To the maintainers: Is my assumption correct that basically the wrong SHA1 hashes are compared?

[darinpantley]

Yes @1100101, I am observing the same behavior with GitLab 10.1.4-ee. Approvals are causing Merge Request events, which are causing my Jenkins job to run. In my case, the repository/commit triggering the event is different than the repository/commit being cloned by the Jenkins job. I do not want approvals to cause my job to run.

I have a Jenkins Pipeline job that gets triggered by GitLab's Jenkins CI integration. The pipeline is defined in pipeline.git, which Jenkins clones automatically. When a Merge Request event happens in projectA.git, the Jenkins job reads environment variables to determine which repository/commit caused the Merge Request event. Then it clones that repository at the specified commit and executes tests against it. The job clones other repositories as well.

I did not notice any environment variables that I can use to abort the job early if it's running due to an approval.

[omehegan]

Sounds like a bug. Sorry for the delay in looking at this. @1100101 can you please capture logs from the plugin when a build is triggered from an MR approval? Look at the User Support section of the README if you need guidance on that. I will look into filtering these.

[1100101]

Hi @omehegan, here's hopefully the relevant log entry.
Due to company policy I had to filter quite a bit of information, I hope that there's still enough for you to work on:

Jan 17, 2018 8:57:21 AM INFO com.dabsquared.gitlabjenkins.webhook.GitLabWebHook getDynamic

WebHook called with url: /project/mw-builder

Jan 17, 2018 8:57:21 AM FINE com.dabsquared.gitlabjenkins.webhook.build.MergeRequestBuildAction

MergeRequest: {
  "object_kind" : "merge_request",
  "user" : {
    "name" : "Frank Aurich",
    "username" : "f.aurich",
    "avatar_url" : ***FILTERED***
  },
  "project" : {
    "name" : "***FILTERED***",
    "description" : "",
    "web_url" : "***FILTERED***",
    "avatar_url" : null,
    "git_ssh_url" : ***FILTERED***,
    "git_http_url" : "***FILTERED***,
    "namespace" : "***FILTERED***",
    "visibility_level" : 10,
    "path_with_namespace" : ***FILTERED***,
    "default_branch" : "develop",
    "homepage" : ***FILTERED***,
    "url" : ***FILTERED***,
    "ssh_url" : ***FILTERED***,
    "http_url" : ***FILTERED***
  },
  "object_attributes" : {
    "id" : 9243,
    "target_branch" : "develop",
    "source_branch" : "defect/9495",
    "source_project_id" : 779,
    "author_id" : 625,
    "assignee_id" : 625,
    "title" : "[DEFECT-9495]: Bugfix: ***FILTERED***",
    "created_at" : "2018-01-09 17:02:51 UTC",
    "updated_at" : "2018-01-17 07:58:23 UTC",
    "milestone_id" : null,
    "state" : "opened",
    "merge_status" : "can_be_merged",
    "target_project_id" : 779,
    "iid" : 805,
    "description" : ***FILTERED***,
    "position" : 0,
    "locked_at" : null,
    "updated_by_id" : 625,
    "merge_error" : null,
    "merge_params" : {
      "force_remove_source_branch" : "1"
    },
    "merge_when_pipeline_succeeds" : false,
    "merge_user_id" : null,
    "merge_commit_sha" : null,
    "deleted_at" : null,
    "in_progress_merge_commit_sha" : null,
    "approvals_before_merge" : null,
    "rebase_commit_sha" : null,
    "lock_version" : 6,
    "time_estimate" : 0,
    "squash" : false,
    "last_edited_at" : "2018-01-16 15:18:21 UTC",
    "last_edited_by_id" : 625,
    "source" : {
      "name" : "***FILTERED***",
      "description" : "",
      "web_url" : "***FILTERED***",
      "avatar_url" : null,
      "git_ssh_url" : ***FILTERED***,
      "git_http_url" : "***FILTERED***",
      "namespace" : "my-project",
      "visibility_level" : 10,
      "path_with_namespace" : ***FILTERED***,
      "default_branch" : "develop",
      "homepage" : ***FILTERED***,
      "url" : ***FILTERED***,
      "ssh_url" : ***FILTERED***,
      "http_url" : "***FILTERED***"
    },
    "target" : {
      "name" : "***FILTERED***",
      "description" : "",
      "web_url" : "***FILTERED***",
      "avatar_url" : null,
      "git_ssh_url" : ***FILTERED***,
      "git_http_url" : "***FILTERED***",
      "namespace" : "my-project",
      "visibility_level" : 10,
      "path_with_namespace" : "",
      "default_branch" : "develop",
      "homepage" : "***FILTERED***",
      "url" : ***FILTERED***,
      "ssh_url" : ***FILTERED***,
      "http_url" : "***FILTERED***"
    },
    "last_commit" : {
      "id" : "d7863e3302b39da75b062b1374b08a8f2a0da472",
      "message" : "Fix issue",
      "timestamp" : "2018-01-16T17:16:50+01:00",
      "url" : "***FILTERED***/commit/d7863e3302b39da75b062b1374b08a8f2a0da472",
      "author" : {
        "name" : ***FILTERED***,
        "email" : ***FILTERED***
      }
    },
    "work_in_progress" : false,
    "total_time_spent" : 0,
    "human_total_time_spent" : null,
    "human_time_estimate" : null,
    "url" : "***FILTERED***/merge_requests/805",
    "action" : "approved"
  },
  "labels" : [ {
    "id" : 2,
    "title" : "Bugfix",
    "color" : "#ff0000",
    "project_id" : 779,
    "created_at" : "2015-10-06 11:45:15 UTC",
    "updated_at" : "2016-06-24 07:00:11 UTC",
    "template" : false,
    "description" : null,
    "type" : "ProjectLabel",
    "group_id" : null
  } ],
  "repository" : {
    "name" : "***FILTERED***",
    "url" : ***FILTERED***,
    "description" : "",
    "homepage" : "***FILTERED***"
  },
  "assignee" : {
    "name" : ***FILTERED***,
    "username" : ***FILTERED***,
    "avatar_url" : "***FILTERED***"
  }
}

Jan 17, 2018 8:57:21 AM INFO com.dabsquared.gitlabjenkins.trigger.handler.AbstractWebHookTriggerHandler handle

mw-builder triggered for merge request.

Two notable items in the JSON are:

"action" : "approved"
and
"merge_status" : "can_be_merged"

[omehegan]

@1100101 yes I see now, it's the "action" : "approved" that we would need to filter. I believe can_be_merged just means there are no merge conflicts.

[omehegan]

Also need to check other possible 'action' values, to filter label changes, edits, reassignments etc.

[1100101]

That’s true. Editing the description of a MR actually also triggers a new build.
Shall I check the logs for this, or do you have an “action” list?

[omehegan]

@1100101 I can't find a doc that explains the possible values of the 'action' field. I also tried grepping through the code but didn't come up with anything. @dblessing can you point me toward anything for this?

[pablobirukov]

GitLab Enterprise Edition 10.1.1-ee

Add label, edit description, and resolve todo produce exactly the same webhook requests (except of correspondingly changed description and values that represent current time). action is set to update everywhere.

There is a field changed in the example, however it's not here, so it's not clear how to filter out such events.

Log

{
    "object_kind": "merge_request",
    "user": {
        "name": "Pavel Birukov",
        "username": "pbirukov",
        "avatar_url": "***FILTERED***"
    },
    "project": {
        "name": "***FILTERED***",
        "description": "***FILTERED***",
        "web_url": "***FILTERED***",
        "avatar_url": "***FILTERED***",
        "git_ssh_url": "***FILTERED***",
        "git_http_url": "***FILTERED***",
        "namespace": "***FILTERED***",
        "visibility_level": 10,
        "path_with_namespace": "***FILTERED***",
        "default_branch": "develop",
        "ci_config_path": null,
        "homepage": "***FILTERED***",
        "url": "***FILTERED***",
        "ssh_url": "***FILTERED***",
        "http_url": "***FILTERED***"
    },
    "object_attributes": {
        "id": 19311,
        "target_branch": "develop",
        "source_branch": "***FILTERED***",
        "source_project_id": 152,
        "author_id": 31,
        "assignee_id": 31,
        "title": "***FILTERED***",
        "created_at": "2018-01-11 16:03:13 UTC",
        "updated_at": "2018-01-19 08:39:33 UTC",
        "milestone_id": null,
        "state": "opened",
        "merge_status": "cannot_be_merged",
        "target_project_id": 152,
        "iid": 10435,
        "description": "",
        "updated_by_id": 31,
        "merge_error": null,
        "merge_params": {
            "force_remove_source_branch": "1"
        },
        "merge_when_pipeline_succeeds": false,
        "merge_user_id": null,
        "merge_commit_sha": null,
        "deleted_at": null,
        "in_progress_merge_commit_sha": null,
        "lock_version": null,
        "time_estimate": 0,
        "last_edited_at": null,
        "last_edited_by_id": null,
        "head_pipeline_id": 65167,
        "ref_fetched": true,
        "merge_jid": null,
        "discussion_locked": null,
        "approvals_before_merge": null,
        "rebase_commit_sha": null,
        "squash": false,
        "source": {
            "name": "***FILTERED***",
            "description": "***FILTERED***",
            "web_url": "***FILTERED***",
            "avatar_url": "***FILTERED***",
            "git_ssh_url": "***FILTERED***",
            "git_http_url": "***FILTERED***",
            "namespace": "***FILTERED***",
            "visibility_level": 10,
            "path_with_namespace": "***FILTERED***",
            "default_branch": "develop",
            "ci_config_path": null,
            "homepage": "***FILTERED***",
            "url": "***FILTERED***",
            "ssh_url": "***FILTERED***",
            "http_url": "***FILTERED***"
        },
        "target": {
            "name": "***FILTERED***",
            "description": "***FILTERED***",
            "web_url": "***FILTERED***",
            "avatar_url": "***FILTERED***",
            "git_ssh_url": "***FILTERED***",
            "git_http_url": "***FILTERED***",
            "namespace": "***FILTERED***",
            "visibility_level": 10,
            "path_with_namespace": "***FILTERED***",
            "default_branch": "develop",
            "ci_config_path": null,
            "homepage": "***FILTERED***",
            "url": "***FILTERED***",
            "ssh_url": "***FILTERED***",
            "http_url": "***FILTERED***"
        },
        "last_commit": {
            "id": "dc9d099b24e2211a31f4bec5a79485491ffdc5e4",
            "message": "***FILTERED***",
            "timestamp": "2018-01-11T18:01:42+02:00",
            "url": "***FILTERED***",
            "author": {
                "name": "Pavel Birukov",
                "email": "***FILTERED***"
            }
        },
        "work_in_progress": false,
        "total_time_spent": 0,
        "human_total_time_spent": null,
        "human_time_estimate": null,
        "url": "***FILTERED***",
        "action": "update"
    },
    "labels": [
        {
            "id": 166,
            "title": "Package",
            "color": "#CCC",
            "project_id": 152,
            "created_at": "2017-07-26 15:33:14 UTC",
            "updated_at": "2017-07-26 15:33:14 UTC",
            "template": false,
            "description": "",
            "type": "ProjectLabel",
            "group_id": null
        }
    ],
    "repository": {
        "name": "***FILTERED***",
        "url": "***FILTERED***",
        "description": "***FILTERED***",
        "homepage": "***FILTERED***"
    },
    "assignee": {
        "name": "Pavel Birukov",
        "username": "pbirukov",
        "avatar_url": "***FILTERED***"
    }
}

[omehegan]

@R00GER thanks, this is helpful. Looks like there are at least two possible values for 'action:' 'update' and 'approved.' I wonder if there is another value for pushes to the MR. I would think that most people would want to build on MR open, pushes to MR, and maybe approvals.

Need some input from @dblessing or someone else at GitLab, unless I can make some time to test other cases.

[1100101]

I've examind the log files from Jenkins to find out which states action can take.
For both changes to the description of the MR and when pushing a new commit to the feature branch, action is set to update.
However, it's still possible to differentiate both states:

Log after updating MR description/settings

[...]
"object_attributes" : {
    "id" : 9561,
    "target_branch" : "develop",
    "source_branch" : *** FILTERED ***,
    "source_project_id" : 779,
    "author_id" : 83,
    "assignee_id" : 83,
    "title" : *** FILTERED ***,
    "created_at" : "2018-01-22 14:41:25 UTC",
    "updated_at" : "2018-01-23 06:26:58 UTC",
    "milestone_id" : null,
    "state" : "opened",
    "merge_status" : "can_be_merged",
    "target_project_id" : 779,
    "iid" : 822,
    "description" : *** FILTERED ***,
    "position" : 0,
    "locked_at" : null,
    "updated_by_id" : 83,
    "merge_error" : null,
    "merge_params" : {
      "force_remove_source_branch" : "1"
    },
    "merge_when_pipeline_succeeds" : false,
    "merge_user_id" : null,
    "merge_commit_sha" : null,
    "deleted_at" : null,
    "in_progress_merge_commit_sha" : null,
    "approvals_before_merge" : 2,
    "rebase_commit_sha" : null,
    "lock_version" : 1,
    "time_estimate" : 0,
    "squash" : true,
    "last_edited_at" : "2018-01-22 14:42:49 UTC",
    "last_edited_by_id" : 83,
    "source" : {
      "name" : *** FILTERED ***,
      "description" : "",
      "web_url" : *** FILTERED ***,
      "avatar_url" : null,
      "git_ssh_url" : *** FILTERED ***,
      "git_http_url" : *** FILTERED ***,
      "namespace" : *** FILTERED ***,
      "visibility_level" : 10,
      "path_with_namespace" : *** FILTERED ***,
      "default_branch" : "develop",
      "homepage" : *** FILTERED ***,
      "url" : *** FILTERED ***,
      "ssh_url" : *** FILTERED ***,
      "http_url" : *** FILTERED ***
    },
    "target" : {
      "name" : *** FILTERED ***,
      "description" : "",
      "web_url" : *** FILTERED ***,
      "avatar_url" : null,
      "git_ssh_url" : *** FILTERED ***,
      "git_http_url" : *** FILTERED ***,
      "namespace" : *** FILTERED ***,
      "visibility_level" : 10,
      "path_with_namespace" : *** FILTERED ***,
      "default_branch" : "develop",
      "homepage" : *** FILTERED ***,
      "url" : *** FILTERED ***,
      "ssh_url" : *** FILTERED ***,
      "http_url" : *** FILTERED ***
    },
    "last_commit" : {
      "id" : "94ef9dc0c6630667e562894dce34f720cb403049",
      "message" : "Fix compiler warnings\n",
      "timestamp" : "2018-01-22T16:08:28+01:00",
      "url" : *** FILTERED ***,
      "author" : {
        "name" : "Frank Aurich",
        "email" : *** FILTERED ***
      }
    },
    "work_in_progress" : false,
    "total_time_spent" : 0,
    "human_total_time_spent" : null,
    "human_time_estimate" : null,
    "url" : *** FILTERED ***,
    "action" : "update"
  },
[...]

Log after pushing new commit

[...]
"object_attributes" : {
    "id" : 9561,
    "target_branch" : "develop",
    "source_branch" : *** FILTERED ***,
    "source_project_id" : 779,
    "author_id" : 83,
    "assignee_id" : 83,
    "title" : *** FILTERED ***,
    "created_at" : "2018-01-22 14:41:25 UTC",
    "updated_at" : "2018-01-23 07:22:09 UTC",
    "milestone_id" : null,
    "state" : "opened",
    "merge_status" : "unchecked",
    "target_project_id" : 779,
    "iid" : 822,
    "description" : *** FILTERED ***,
    "position" : 0,
    "locked_at" : null,
    "updated_by_id" : 83,
    "merge_error" : null,
    "merge_params" : {
      "force_remove_source_branch" : "1"
    },
    "merge_when_pipeline_succeeds" : false,
    "merge_user_id" : null,
    "merge_commit_sha" : null,
    "deleted_at" : null,
    "in_progress_merge_commit_sha" : null,
    "approvals_before_merge" : 2,
    "rebase_commit_sha" : null,
    "lock_version" : 1,
    "time_estimate" : 0,
    "squash" : true,
    "last_edited_at" : "2018-01-22 14:42:49 UTC",
    "last_edited_by_id" : 83,
    "source" : {
      "name" : *** FILTERED ***,
      "description" : "",
      "web_url" : *** FILTERED ***,
      "avatar_url" : null,
      "git_ssh_url" : *** FILTERED ***,
      "git_http_url" : *** FILTERED ***,
      "namespace" : *** FILTERED ***,
      "visibility_level" : 10,
      "path_with_namespace" : *** FILTERED ***,
      "default_branch" : "develop",
      "homepage" : *** FILTERED ***,
      "url" : *** FILTERED ***,
      "ssh_url" : *** FILTERED ***,
      "http_url" : *** FILTERED ***
    },
    "target" : {
      "name" : *** FILTERED ***,
      "description" : "",
      "web_url" : *** FILTERED ***,
      "avatar_url" : null,
      "git_ssh_url" : *** FILTERED ***,
      "git_http_url" : *** FILTERED ***,
      "namespace" : *** FILTERED ***,
      "visibility_level" : 10,
      "path_with_namespace" : *** FILTERED ***,
      "default_branch" : "develop",
      "homepage" : *** FILTERED ***,
      "url" : *** FILTERED ***,
      "ssh_url" : *** FILTERED ***,
      "http_url" : *** FILTERED ***
    },
    "last_commit" : {
      "id" : "7ab6bbdf213e6a7fd786bd672b42bf8b1b4dee7d",
      "message" : "Merge remote-tracking branch 'origin/develop' into *** FILTERED ***\n",
      "timestamp" : "2018-01-23T08:20:48+01:00",
      "url" : *** FILTERED ***,
      "author" : {
        "name" : "Frank Aurich",
        "email" : *** FILTERED ***
      }
    },
    "work_in_progress" : false,
    "total_time_spent" : 0,
    "human_total_time_spent" : null,
    "human_time_estimate" : null,
    "url" : *** FILTERED ***,
    "action" : "update",
    "oldrev" : "94ef9dc0c6630667e562894dce34f720cb403049"
  },
[...]

In the latter case, an additional JSON value oldrev is available right after action.
This would make it possible to discern between the two cases.

[1100101]

It seems issue #659 is the opposite of my issue, but regarding the approved action, the problem might actually be gandled by issue #664? I see that that MR introduces an option to build when an MR gets approved.

I'm currently not able to verify this by installing the latest plugin version, unfortunately :/

[omehegan]

@1100101 I don't have a way to test EE features, so I can't reproduce your issue or test the fix. Not sure how to proceed with this. Actually the fix in #664 has not been released yet. Would you be able to test with a patch release?

[jakub-bochenski]

@omehegan I was able to reproduce an issue with Gitlab 10.4.3-ee and and the plugin build from a2b8394

A build is triggered when a MR is approved even though the build is configured with <triggerOnApprovedMergeRequest>false</triggerOnApprovedMergeRequest>

I will try to find the root cause, there are however two strange things.

First: I was not able to reproduce the above issue previously so there might be some incompatible hook changes between Gitlab versions.

Second: this issue was reported before any of my PRs were even opened (not to mention merged). There has to be some other factor at work here. Maybe it would also explain why I was unable to reproduce the issue before.

[omehegan]

This was fixed/released in version 1.5.4 of the plugin, which should be available in the update center soon.

[jkugler]

This bug appears to be back. Using GitLab Plugin 1.5.11. Same configuration as discussed: Push Events no checked, "Merge request events" checked.

[anapsix]

as I can see, Gitlab is sending "event_type": "merge_request" when MR description changes..
Even though changes contains only description related updates, build is triggered..
imho, plugin should ignore non-code changes and not trigger a build, with exception of comment-based trigger phrase

[1100101]

@anapsix this issue is more related to #871 than this one (here the issue was specifically with build triggers on approval).

[trunov-ms]

I have several Jenkins jobs with parallel executing on Push Events, Tag Push Events and MR Events. Gitlab-plugin settings are "Open MR" and "Rebuild on push to source branch".
These jobs sets a label in MR depending on the build result.
After MR labels was changed, more longer jobs triggerring again although builds with this changes are still executing.

[seagle0128]

Same issue happens again, with the latest version.

[nbauma109]

I have the same issue:

• Gitlab plugin 1.5.9
• Gitlab EE 13.6.7-ee

See also my comment on #796