Upgrading to Slack 2.0 outbound webhooks breaks Jenkins [replicated] #191

samrocketman · 2016-03-16T22:52:06Z

This issue is going to be the center point for tracking the patching progress. This issue supersedes the following issues:

This issue was caused by the following merged change: #160

Workaround (for Jenkins admins)

There's two options for a workaround.

Option 1

See #191 (comment) which will remove all slack configurations and give an opportunity to reconfigure your Jenkins with the latest update of the Slack plugin.

Option 2

This will entirely disable outbound webhooks in the Slack plugin 2.0.

Go to your $JENKINS_HOME and create a init.groovy.d/ directory.
Download this script to disable Slack outbound webhooks and copy it into $JENKINS_HOME/init.groovy.d.
Restart your Jenkins instance.

It should be back to normal and you'll be able to configure things again. Slack outbound webhooks will always be disabled for the duration you have this script installed in init.groovy.d/.

Problematic conditions

I was able to successfully reproduce the issue under the following conditions.

Another plugin with webhooks configured (the bootstrap includes ghprb plugin i.e. github pull request builder plugin)
Authentication enabled (recommend using the pre-installed github oauth plugin from the bootstrap).
Only authenticated users get access (i.e. not anonymous). I simply set the authenticated group to be administrators with the Administer permission.
Upgrading from Slack 1.8.1 to 2.0.0.

How to reproduce (for developers to patch)

Bootstrap Jenkins with slack configured. Simply clone the repository and execute ./slack_bootstrap.sh on a Unix-like machine. This will:
- Pre-install an older version of Jenkins.
- Pre-install and configure the slack plugin 1.8.1.
- Configure some test jobs with slack configured.
- Pre-install github-oauth and ghprb plugin.
Configure global security:
1. Enable global security.
2. Generate a new GitHub OAuth Application. With the following settings:
  - Application name: Jenkins
  - Homepage URL: http://localhost:8080/
  - Application description: Jenkins testing.
  - Authorization callback URL: http://localhost:8080/securityRealm/finishLogin
3. Security Realm should be set to Github Authentication Plugin and configure it with the Client ID and Client Secret settings from your GitHub OAuth application.
4. Under access control in the global security page configure Authorization and choose Matrix-based security. Set the following settings:
  - authenticated group must have administer permissions.
  - anonymous group must not have any permissions.
Visit http://localhost:8080/manage and update Jenkins to latest edition.
Restart Jenkins after upgrade.
Visit http://localhost:8080/pluginManager/ and update all plugins (including Slack plugin 2.0) to latest editions.
Restart Jenkins after upgrade.

From this point, you should get a stack trace like in JENKINS-33556. Following the above steps reproduces the issue every time.

Additional bootstrap commands:

Restart the Jenkins instance with ./scripts/provision_jenkins.sh restart
Delete and clean up everying. ./gradlew clean
Bootstrap everything again. ./slack_bootstrap.sh

The text was updated successfully, but these errors were encountered:

kmadel · 2016-03-17T09:46:16Z

I understand the issue with #189 - but not clear how the webhook changes caused #188
I couldn't reproduce #188 but did see #189

samrocketman · 2016-03-22T06:39:05Z

Re-opening since there's a report stating it's still not fixed. I'll have to follow up with my own testing tomorrow.

daniel-beck · 2016-03-22T09:15:16Z

@samrocketman Why is the webhook URL even configurable in #160, rather than (maybe) having a checkbox for enable/disable? Seems unnecessary, and I've never seen a plugin do this.

samrocketman · 2016-03-22T21:58:08Z

@daniel-beck not sure, for the most part I've largely handed off this project to @kmadel as I don't do much development for this plugin. Even when I did maintain it I didn't actually develop. I only merged others' pull requests. I wasn't really involved in the merge of #160 but when things broke I did provide a workaround to disable it. I agree it should have an option to be disabled.

Side note, I handed it off because I don't/didn't actually use the slack plugin and I figure if someone is actually using it they'll be able to better support it.

OwensCode · 2016-03-22T23:40:14Z

I wasn't involved in the development of the feature but I was involved with the code review. I recommended that the URL be configurable so you could run it alongside other similar plugins without clashing, especially since it was originally listening on /webhook. A checkbox to enable/disable sounds like a good enhancement. It also works around another concern that I had, which is that it was enabled by default, even if you have no interest in the functionality and could consider it a security risk.

netbe · 2016-03-23T11:30:39Z

@OwensCode any idea how I can disable this once installed since jenkins is crashed, a quick fix could be to give the process to deactivate it if that's the problem + deactivate it by default for next release

OwensCode · 2016-03-23T12:52:56Z

@netbe There's a link to a script by @samrocketman at the top of this page, in the "Workaround (for Jenkins admins)" section.

OwensCode · 2016-04-15T17:59:02Z

Is this still an issue @samrocketman? Do we have any more information from people still being impacted by it after the fix was released? I would like to take a look at fixing it, if there is something to fix.

samrocketman · 2016-04-15T18:46:20Z

@OwensCode let me take a look this evening or weekend. I'll follow up and try to be detailed if I can reproduce.

mikz · 2016-05-05T09:15:14Z

This was issue for me. Fresh installation of slack plugin on jenkins 2.1. And even though I set up the outgoing webhooks, the Global Tool Configuration crashed pointing to

slack-plugin/src/main/java/jenkins/plugins/slack/webhook/WebhookEndpoint.java

Line 51 in ae8e6e3

String url = globalConfig.getSlackOutgoingWebhookURL();

.

The workaround worked and the page no longer crashes.

pkkummermo · 2016-07-13T09:41:28Z

I had Jenkins failing during startup with the same issues as above. My solution was the following:

Remove $JENKINS_HOME/plugins/slack
Restart Jenkins
Go to Manage Jenkins -> Manage Old Data -> Discard old data
Install Slack Plugin
Jenkins restart.

CAUTION: This will wipe your old Slack configurations, but I prioritized having the Slack integration up and running and took the time to redo the integrations.

ElaineRichards · 2016-07-19T16:15:30Z

My workaround to get Jenkins back is to move the jobs directory to "jobs.safe", restart jenkins, uninstall the Slack plugin, remove the generated empty jobs directory and move "jobs.safe"

The side effect of this is that I've had to change my Perforce population option from "forced clean and sync" to "Auto cleanup".

I reluctantly reinstalled the plugin yesterday (so the version was the latest as of July 18) because my coworkers really want the slack notifications again. However, someone must have restarted Jenkins in the middle of the night because I came in to another dead Jenkins server. So, it's staying off.

Here's the latest stack trace:

java.lang.NullPointerException
at jenkins.plugins.slack.webhook.WebhookEndpoint.getUrlName(WebhookEndpoint.java:51)
at jenkins.model.Jenkins.getUnprotectedRootActions(Jenkins.java:4342)
at jenkins.model.Jenkins.getTarget(Jenkins.java:4311)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:674)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

samrocketman · 2016-07-19T19:28:23Z

@ElaineRichards see the workaround discussed in the description of this issue.

SimonPNorra · 2016-09-09T09:17:04Z

Please promote the solution of @pkkummermo from 13th of July.

This did work for me as well.

Thanks @pkkummermo

samrocketman · 2016-09-10T00:40:26Z

@SimonPNorra

Please promote the solution of @pkkummermo from 13th of July.

This did work for me as well.

Thanks @pkkummermo

I updated the original post to reflect his solution as "Option 1".

pedromartinez11 · 2017-06-15T00:20:00Z

Does 2.2 have the same problems?

On 2.1 I'm able to go into Jenkins, but I have to regularly restart it after getting the NullPointerException from the slack plugin.

j796160836 · 2017-07-22T03:58:14Z

I have installed lastest slack notification plugins (version 2.2) also encountered this error.
Apply Option 2 Workaround (see above) and save my time, thanks.

fengxx · 2018-01-13T02:44:48Z

to workaround the issue, run the script on Jenkins script console

import jenkins.plugins.slack.webhook.GlobalConfig;
import jenkins.plugins.slack.webhook.WebhookEndpoint;
def splackConfig=Jenkins.getActiveInstance().getDescriptor(GlobalConfig.class);
def slackWebHook=Jenkins.getActiveInstance().getExtensionList(WebhookEndpoint.class).get(0);
println splackConfig
println slackWebHook
println slackWebHook.@globalConfig
slackWebHook.@globalConfig=splackConfig

It will output something like
jenkins.plugins.slack.webhook.GlobalConfig@526ba2e2
jenkins.plugins.slack.webhook.WebhookEndpoint@15ab7a57
null
Result: jenkins.plugins.slack.webhook.GlobalConfig@526ba2e2

boxymoron · 2018-04-06T21:38:19Z

This is still happening on Jenkins ver. 2.114 + Slack Notification Plugin ver. 2.3
The fix from @fengxx worked for these versions.

brandoncordell · 2018-09-14T19:20:13Z

This is still happening. Is this plugin abandoned? Is there another plugin I should use instead?

hugespoon · 2018-10-01T17:05:54Z

Can confirm that this issue still seems to be occurring (Jenkins 2.32.2 and Slack Plugin 2.3).

Lots of the following in the logs:

at jenkins.plugins.slack.webhook.WebhookEndpoint.getUrlName(WebhookEndpoint.java:51) at jenkins.model.Jenkins.getDynamic(Jenkins.java:3565)

Flightkick · 2018-11-14T10:50:39Z

At the time of writing (plugin version 2.3) this is still an issue.
A reboot of the Jenkins instance will fix the NPE being thrown.

@Wadeck 's pull request appears to fix the issue but the fix has not yet been released: #403

timja · 2018-11-28T15:26:51Z

Closing in favour of merged fix, will be released soon

samrocketman added bug HIGH labels Mar 16, 2016

This was referenced Mar 16, 2016

FATAL: class jenkins.plugins.slack.SlackNotifier$SlackJobProperty is missing its descriptor since 2.0 #188

Closed

Version 2.0 seems to break GitLab plugin webhook triggers #189

Closed

samrocketman changed the title ~~Slack outbound webhooks breaks upgrade [replicated]~~ Upgrading to Slack 2.0 outbound webhooks breaks Jenkins [replicated] Mar 16, 2016

samrocketman added this to the 2.0.1 milestone Mar 17, 2016

samrocketman mentioned this issue Mar 17, 2016

Set webhook endpoint to random uuid based string if not set #190

Merged

kmadel closed this as completed in 585494d Mar 17, 2016

This was referenced Mar 17, 2016

2.0.1 Release #193

Closed

prepare for release 2.0.1 #194

Merged

samrocketman mentioned this issue Mar 17, 2016

Slack plugin 2.0 breaks Jenkins 1.647 #196

Closed

daniel-beck mentioned this issue Mar 21, 2016

Refine the plugin selection jenkinsci/jenkins#2152

Merged

samrocketman reopened this Mar 22, 2016

samrocketman mentioned this issue Mar 22, 2016

Slack-plugin makes jenkins crash #199

Closed

samrocketman added MEDIUM and removed MEDIUM labels Mar 29, 2016

samrocketman mentioned this issue Jun 11, 2016

NullPointerException for Outgoing slack hook #226

Closed

samrocketman mentioned this issue Nov 23, 2016

slackSend Pipeline DSL method missing after upgrade to Jenkins 2.9 #229

Closed

samrocketman mentioned this issue Mar 11, 2017

Slack plugin completely burning my Jenkins environment #301

Closed

gabemontero mentioned this issue Jun 23, 2017

Jenkins error after session timeout openshift/jenkins-openshift-login-plugin#16

Closed

gala-dan mentioned this issue Dec 18, 2017

Add Slack integration into build process galatea-associates/fuse-starter-java#100

Closed

fengxx mentioned this issue Jan 13, 2018

fix NPE on fresh new install #364

Closed

bradrhod mentioned this issue Feb 26, 2018

NullPointerException on many manage jenkins pages from unconfigured Slack plugin #369

Closed

Wadeck mentioned this issue Oct 4, 2018

[JENKINS-37339] Prevent NPE after plugin installation #403

Merged

timja closed this as completed Nov 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrading to Slack 2.0 outbound webhooks breaks Jenkins [replicated] #191

Upgrading to Slack 2.0 outbound webhooks breaks Jenkins [replicated] #191

samrocketman commented Mar 16, 2016 •

edited

Loading

kmadel commented Mar 17, 2016

samrocketman commented Mar 22, 2016

daniel-beck commented Mar 22, 2016

samrocketman commented Mar 22, 2016

OwensCode commented Mar 22, 2016

netbe commented Mar 23, 2016

OwensCode commented Mar 23, 2016

OwensCode commented Apr 15, 2016

samrocketman commented Apr 15, 2016

mikz commented May 5, 2016

pkkummermo commented Jul 13, 2016

ElaineRichards commented Jul 19, 2016 •

edited

Loading

samrocketman commented Jul 19, 2016

SimonPNorra commented Sep 9, 2016

samrocketman commented Sep 10, 2016

pedromartinez11 commented Jun 15, 2017

j796160836 commented Jul 22, 2017 •

edited

Loading

fengxx commented Jan 13, 2018

boxymoron commented Apr 6, 2018

brandoncordell commented Sep 14, 2018

hugespoon commented Oct 1, 2018

Flightkick commented Nov 14, 2018

timja commented Nov 28, 2018

Upgrading to Slack 2.0 outbound webhooks breaks Jenkins [replicated] #191

Upgrading to Slack 2.0 outbound webhooks breaks Jenkins [replicated] #191

Comments

samrocketman commented Mar 16, 2016 • edited Loading

Workaround (for Jenkins admins)

Option 1

Option 2

Problematic conditions

How to reproduce (for developers to patch)

kmadel commented Mar 17, 2016

samrocketman commented Mar 22, 2016

daniel-beck commented Mar 22, 2016

samrocketman commented Mar 22, 2016

OwensCode commented Mar 22, 2016

netbe commented Mar 23, 2016

OwensCode commented Mar 23, 2016

OwensCode commented Apr 15, 2016

samrocketman commented Apr 15, 2016

mikz commented May 5, 2016

pkkummermo commented Jul 13, 2016

ElaineRichards commented Jul 19, 2016 • edited Loading

samrocketman commented Jul 19, 2016

SimonPNorra commented Sep 9, 2016

samrocketman commented Sep 10, 2016

pedromartinez11 commented Jun 15, 2017

j796160836 commented Jul 22, 2017 • edited Loading

fengxx commented Jan 13, 2018

boxymoron commented Apr 6, 2018

brandoncordell commented Sep 14, 2018

hugespoon commented Oct 1, 2018

Flightkick commented Nov 14, 2018

timja commented Nov 28, 2018

samrocketman commented Mar 16, 2016 •

edited

Loading

ElaineRichards commented Jul 19, 2016 •

edited

Loading

j796160836 commented Jul 22, 2017 •

edited

Loading