Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to configure with icinga-director #5

Closed
lcorsini opened this issue Jun 20, 2017 · 40 comments
Closed

how to configure with icinga-director #5

lcorsini opened this issue Jun 20, 2017 · 40 comments

Comments

@lcorsini
Copy link

lcorsini commented Jun 20, 2017

I've tried installing this component, but I'm stuck, since I'm using icinga-director.
I've installed and configured the deb package.
In icinga-director added the vars.slack_notifications to my hosts, but when I apply the config it says
critical/config: Error: Import references unknown template: 'slack-notifications-user-configuration-hosts' Location: in /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf: 47:3-47:55 /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(45): /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(46): apply Notification "slack-notifications-notification-hosts" to Host { /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(47): import "slack-notifications-user-configuration-hosts" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
because director doesn't know anything about this new template

@staenker
Copy link
Collaborator

I'll get to it this Friday,
cheers,
Richard

@lcorsini
Copy link
Author

Hi, any update on this?

@majales
Copy link

majales commented Dec 8, 2017

Hi.. I did it on Director. I created Data Field slack_notifications and added it to host-template in Director and set it to "enabled" .. and added it to service-template in Director also with "enabled" .. It did the trick.. I had not modified hosts and service template file,s it is not necessary.. It was on icinga 2.7 and director 1.3.1 ( slightly newer then release.. It was taken from github repo in April )

@rgevaert
Copy link

rgevaert commented Dec 8, 2017

Same here!

@flaviomeyer
Copy link

Hi @nisabek

Thanks for this nice notification plugin. I'm looking Icinga Director install guide. Is it possible to create a guide for Icinga Director?

@majales Maybe you can post your solution?

Thanks :)

Flavio

@nisabek
Copy link
Owner

nisabek commented Mar 1, 2018

Hi all,

Thanks for addressing this, and sorry for late response.
I just got to try icinga-director, and in order for the director to recognize the changes in the icinga core I run the command icingacli director kickstart run

That should be run after the usual installation and final restart of the icinga2 service.
After that the slack-notifications-command is visible in the command list of the icinga director, and if your hosts have the vars.slack_notifications = "enabled" set it will fire the notifications.

@lcorsini @meyerf99 - I would appreciate if you give it a try and report feedback.

Thanks and Regards,
Nune

@flaviomeyer
Copy link

flaviomeyer commented Mar 1, 2018

Hey @nisabek

Thanks for your fast answer. I got this error with timeperiod.

critical/config: Error: Validation failed for object 'x.ch!disk_check!slack-notifications-notification-services' of type 'Notification'; Attribute 'period': Object '24x7' of type 'TimePeriod' does not exist. Location: in /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf: 8:5-8:19 /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(6): user_groups = [ "icingaadmins" ] /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(7): /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(8): period = "24x7" ^^^^^^^^^^^^^^^ /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(9): /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(10): interval = 5m
Maybe you know how i can fix it?

Kind regards,
Flavio

@rgevaert
Copy link

rgevaert commented Mar 1, 2018

It's possible to manage the custom attributes on host/service level through director. However configuring the other things isn't possible. Or at least I didn't find how. I added them to the configuration files.

@flaviomeyer
Copy link

flaviomeyer commented Mar 1, 2018

I've fixed my problem, but it was very hard. @nisabek Maybe u can documentation with timeperiod and icingaadmins group as prerequisite. Is it possible to test notifications?

From my point of view, I have everything set up, but unfortunately I think no message in Slack.
Does the script have a logfile? If yes, where?

Kind regards,

Flavio

PS: In my Debug Log i see only this:
[2018-03-01 20:58:04 +0100] debug/slack-notifications: Sending notification...read user configuration successfully: {"vars.slack_notifications_botname":null,"vars.slack_notifications_channel":null,"vars.slack_notifications_icinga2_base_url":null,"vars.slack_notifications_icon_dictionary":null,"vars.slack_notifications_plugin_output_max_length":0.0,"vars.slack_notifications_webhook_url":null}

I think thats false, coz all infos have null as string, is that right?

@lcorsini
Copy link
Author

lcorsini commented Mar 2, 2018

In the meantime, I've actually reinstalled my infrastructure without director so I cannot really test this anymore in the short term.

@flaviomeyer
Copy link

flaviomeyer commented Mar 2, 2018

@nisabek I've fixed my problem.

Set in Icinga Director Notification Interval to 60 and not 0.
image

See this link in: https://monitoring-portal.org/woltlab/index.php?thread/41134-slack-notification-im-director-per-user-zuweisung/

@nisabek
Copy link
Owner

nisabek commented Sep 8, 2018

seems like this is fixed. The current master contains the reference to a docker image that can be used for testing.

@nisabek nisabek closed this as completed Sep 8, 2018
@nisabek nisabek reopened this Jun 18, 2019
@nisabek
Copy link
Owner

nisabek commented Jun 18, 2019

It looks like it's still unclear how to configure the plugin in Icinga Director.
@majales @rgevaert - could you clarify if you had any luck configuring the webhook_url and making it work in Icinga Director? If so, I'd be very thankful for a README contribution with steps how to achieve that.

Thanks!
Nune

@rgevaert
Copy link

Sorry, I'm not using icinga2 anymore (different job) so can't help unfortunately.

@buggsdummy
Copy link

Can someone update this? Thanks!

@flaviomeyer
Copy link

Interesting...
I have looked for this notification addon and found out that is not working anymore.
Maybe I build in the next few days a new slack notification plugin which works with Icinga Director.

@NC-Yungd
Copy link

After running the kickstart wizard in Icinga2 Director, is this the expected output for the NotificationCommand?

image

I am unable to get any messages in my Slack channel. Below are from my debug logs:

# tail -f /var/log/icinga2/debug.log | grep "warning/PluginNotificationTask\|slack-notifications"
[2019-12-12 17:45:15 +0000] notice/Notification: Attempting to send notifications of type 'Custom' for notification object 'server1.com!slack-notifications-notification-hosts'.
[2019-12-12 17:48:22 +0000] notice/NotificationComponent: Attempting to send reminder notification 'server2.com!linux_apt!slack-notifications-notification-services'.
[2019-12-12 17:48:22 +0000] notice/Notification: Attempting to send reminder notifications of type 'Problem' for notification object 'server3.com!linux_apt!slack-notifications-notification-services'.
[2019-12-12 17:48:22 +0000] notice/NotificationComponent: Attempting to send reminder notification 'server4.com!linux_apt!slack-notifications-notification-services'.

@nisabek
Copy link
Owner

nisabek commented Dec 12, 2019

@NC-Yungd can't verify right now, if that's the expected output, but I can tell you that those logs are not from the plugin(your grep is correct, just it matches some other lines...)...are you sure everything is enabled on the plugin side?

@flaviomeyer
Copy link

@NC-Yungd A few weeks ago I have rebuilt the Slack notification. If I find time tomorrow I'il create a new gist and I'il let you know.

@NC-Yungd
Copy link

Here are my current configurations:

/etc/icinga2/zones.conf

object Endpoint "icinga2.xyz.com" {
}

object Zone "master" {
	endpoints = [ "icinga2.xyz.com" ]
}

object Zone "global-templates" {
	global = true
}

object Zone "director-global" {
	global = true
}

/etc/icinga2/zones.d# tree -d

.
├── global-templates
│   ├── commands
│   └── templates
└── master
    ├── archive
    └── slack-notifications

/etc/icinga2/zones.d/master/slack-notifications# cat slack-notifications-user-configuration.conf

template Notification "slack-notifications-user-configuration" {
    import "slack-notifications-default-configuration"

    vars.slack_notifications_webhook_url = "https://hooks.slack.com/services/secret1/secret2"
    vars.slack_notifications_icinga2_base_url = "http://icinga2.xyz.com"
    vars.slack_notifications_channel = "#icinga2_alerts"
    vars.slack_notifications_botname = "icinga2_bot"
}

template Notification "slack-notifications-user-configuration-hosts" {
    import "slack-notifications-default-configuration-hosts"
}

template Notification "slack-notifications-user-configuration-services" {
    import "slack-notifications-default-configuration-services"
}

In Icinga2 Director, I've created the custom variable for slack_notifications as a string. Afterwards, I've add ed slack_notifications as a field to my Host and Services template and included the string "enabled".

image

image

@nisabek
Copy link
Owner

nisabek commented Dec 13, 2019

hmm, from the fact that there are no logs of the plugin, I suppose that the slack_notifications variable is not understood by the director...
when you say "custom variable for slack_notifications as a string" do you mean a data field? Cause @majales mentioned creating a data_field and I'm not sure if that's the same as "custom variable"

Sorry I can't do more at the moment :/

@NC-Yungd
Copy link

@nisabek Initially, I installed this plugin by using the "Debian package" method, but after installing with git and copying the slack-notification conf files into my Icinga2 zones.d/master folder, I was able to get notifications in Slack.

I'm going to assume that the Debian package is out-of-date since I wasn't able to get Slack notifications with Director.

@nisabek
Copy link
Owner

nisabek commented Dec 15, 2019

Thanks @NC-Yungd for getting back to us and making it work!
I'm having a bit of a trouble updating the dbian repo, in the meantime I updated the README

https://github.com/nisabek/icinga2-slack-notifications#installation-using-debian-package
https://github.com/nisabek/icinga2-slack-notifications#running-with-icinga-director

If you feel like there's information missing regarding making it work on icinga director, I'd appreciate a a PR with more details.

Thanks!
Nune

Edit: updated wrong links.

@guzaho
Copy link

guzaho commented Mar 27, 2020

Hi there!

Thanks for creating such a useful plugin.

I recently installed icinga2 with icingaweb2 and director, and I need to integrate is to my company's slack.

I've followed the howto and installed from git (and discovered a few mistakes in the howto), as suggested, but I cannot make it work.

This is what I have done:

ONE) Created a slack webhook and verified with curl that I do indeed get messages into the right channel.

TWO) Logged into the icinga2 master machine as root, and

cd /etc/icinga2/conf.d/
git clone https://github.com/nisabek/icinga2-slack-notifications.git
cd icinga2-slack-notifications
cp src/slack-notifications/slack-notifications-user-configuration.conf.template slack-notifications-user-configuration.conf
chown -R nagios:nagios /etc/icinga2/conf.d

THREE) Customized /etc/icinga2/conf.d/templates.conf

template Host "generic-host" {
max_check_attempts = 3
check_interval = 1m
retry_interval = 30s

check_command = "hostalive"
vars.slack_notifications = "enabled"
}

/**
* Provides default settings for services. By convention
* all services should import this template.
*/
template Service "generic-service" {
max_check_attempts = 5
check_interval = 1m
retry_interval = 30s

vars.slack_notifications = "enabled"
}

FOUR) Customized /etc/icinga2/conf.d/icinga2-slack-notifications/slack-notifications-user-configuration.conf

template Notification "slack-notifications-user-configuration" {
import "slack-notifications-default-configuration"

vars.slack_notifications_webhook_url = "https://hooks.slack.com/services/secret1/secret2"
vars.slack_notifications_icinga2_base_url = "https://my.company.com/icingaweb2"
vars.slack_notifications_channel = "#service-health"
vars.slack_notifications_botname = "service_health_bot"
}

template Notification "slack-notifications-user-configuration-hosts" {
import "slack-notifications-default-configuration-hosts"
}

template Notification "slack-notifications-user-configuration-services" {
import "slack-notifications-default-configuration-services"
}

FIVE) Enable Icinga2 debug logging
icinga2 feature enable debuglog

SIX) Restarted icinga2
systemctl restart icinga2

SEVEN) Triggered an alert by stopping a service, and verified that icingaweb2 shows the alert

But I don't see any slack in the debug file.
grep "warning/PluginNotificationTask\|slack-notifications" /var/log/icinga2/debug.log

I have not configured anything in the director, because I'm new to Icinga2, and the discussions are not clear on what to do in what order.

I don't have zones configured, do I need to?

Can anyone please help?

@nisabek
Copy link
Owner

nisabek commented Mar 27, 2020

hi @guzaho and thanks for using the plugin
It's getting hard to solve these issues, since everyone's setup looks different, but I'm gonna try.

question - does the service you stopped to test, use the generic-service template?...
Can you show how it is configured?...
another question - Do you see any logs in the debug.log?..

@nisabek nisabek changed the title how to configure with icinga-director [Help wanted] how to configure with icinga-director Mar 27, 2020
@nisabek nisabek changed the title [Help wanted] how to configure with icinga-director how to configure with icinga-director Mar 27, 2020
@guzaho
Copy link

guzaho commented Mar 27, 2020

Hi @nisabek !

Thanks for the prompt reply!

I do see plenty of logs in the debug.log.

Actually, right before I saw your message now, I got a message in slack! Yay!

It seems however, that I am only getting messages for the master node.

The service I stopped is checked by check_systemd, and my service template looks like this in director:

zones.d/director-global/service_templates.conf
template Service "check_systemd" {
check_command = "check_systemd"
command_endpoint = host_name
vars.slack_notification = "enabled"
}

The service set that uses the service template is like this:

apply Service "check_systemd" {
import "check_systemd"

assign where "sentinel-template" in host.templates

import DirectorOverrideTemplate

}

So I guess it does not use the generic-service template?

@nisabek
Copy link
Owner

nisabek commented Mar 27, 2020

oh wait, that's a template on it's own, but do I see a typo?...
vars.slack_notifications = "enabled"

@guzaho
Copy link

guzaho commented Mar 27, 2020

ok, fixed the missing s

zones.d/director-global/service_templates.conf
template Service "check_systemd" {
check_command = "check_systemd"
command_endpoint = host_name
vars.slack_notifications = "enabled"
}

@nisabek
Copy link
Owner

nisabek commented Mar 27, 2020

and?...works?.. :)

@guzaho
Copy link

guzaho commented Mar 27, 2020

No, not for other hosts.. just the master... which does only use default services like apt, procs, etc. My nodes only use check_apt and check_procs, etc.

The particular service I'm testing does not post any msgs to slack, it seems...

@guzaho
Copy link

guzaho commented Mar 27, 2020

I've followed the icinga2 official docs to the point. I've set up service templates, then service groups and the assigned that to hosts. So any service that inherits the template for check_systemd has now gotten the slack_notifications set to "enabled":

image

image

image

@nisabek
Copy link
Owner

nisabek commented Mar 27, 2020

alrighty that looks correct...questions

  1. you say, you get notifications from your 'master' node - which template does the master node use?..
    on a side note - you must be then seeing logs from the master node...
  2. in the comment by @NC-Yungd it is mentioned that
    "In Icinga2 Director, I've created the custom variable for slack_notifications as a string. Afterwards, I've add ed slack_notifications as a field to my Host and Services template and included the string "enabled"."

can you try that?...

@guzaho
Copy link

guzaho commented Mar 27, 2020

Of course, I can try! Thank you!

  1. Yes, I do see slack messages in the debug log now, but only for the master node.

I am not sure which template the master node uses, it just showed up after installing icinga2, icingaweb2 and the director.. How can I check?

  1. Yes, I did as @NC-Yungd suggested. I've created the slack_notifications as a string in data fields:

image

image

I have now also set the slack_notifications to enabled for the host, and it works!!
image

@guzaho
Copy link

guzaho commented Mar 27, 2020

Ok, so just to sum up, the missing piece what that misspelling that check_notifications should be in plural form, and that the hosts did not have the slack_notifications set to "enabled"!

thank you so much!

@nisabek
Copy link
Owner

nisabek commented Mar 27, 2020

I looked through the docs again, and your configuration, and I don't seem to find if you have this piece somewhere

apply Notification "slack-notifications-notification-services" to Service {
  import "slack-notifications-user-configuration-services"

  assign where service.vars.slack_notifications == "enabled"
}

@guzaho
Copy link

guzaho commented Mar 27, 2020

Hm, that's true. Where does that go?

@nisabek
Copy link
Owner

nisabek commented Mar 27, 2020

ah, ignore that, that's in the source code! so you must have checked it out :D my memory of the project is not that fresh sorry :)

@nisabek
Copy link
Owner

nisabek commented Mar 27, 2020

so - works now?

@guzaho
Copy link

guzaho commented Mar 27, 2020

Yes, looks like it. I'm gonna do some more testing, just in case. But all good for now! Thank you so much. 😄

@nisabek
Copy link
Owner

nisabek commented Mar 27, 2020

awesome, enjoy!

I really want to close this issue cause it's turned into debugging session of different issues.

Anybody having trouble with icinga2 director configuration please refer to last part of the docs
https://github.com/nisabek/icinga2-slack-notifications#running-with-icinga-director

If that doesn't help - please open a new issue. And of course, contribute - with more docs, more tests, ...! ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants