Description

Installs and configures Icinga 1.7 for a server and nagios-plugins for clients with ssh for plugin execution, using Chef search capabilities. This cookbook is based on Opscode's nagios cookbook, but is modified to fit the environments I more typically deploy (icinga under lighttpd, PAM auth, etc.) The changes are not yet complete.

Requirements

Chef

Chef version 0.10.0+ is required for chef environment usage. See Environments under Usage below.

A data bag named 'users' should exist, see Data Bag below.

The monitoring server that uses this recipe should have a role named 'monitoring' or similar, this is settable via an attribute. See Attributes below.

Because of the heavy use of search, this recipe will not work with Chef Solo, as it cannot do any searches without a server.

By default NRPE clients can only be monitored by Nagios servers in the same environment. To change this set the multi_environment_monitoring attribute. See Attributes below.

Platform

Debian 6
Ubuntu 10.04, 12.04
Red Hat Enterprise Linux (CentOS) 5.8, 6.3

Notes: This cookbook has been tested on the listed platforms. It may work on other platforms with or without modification.

Cookbooks

apache2
build-essential
php
nginx
nginx_simplecgi

Attributes

default

The following attributes are used by both client and server recipes.

node['icinga']['user'] - icinga user, default 'nagios'.
node['icinga']['group'] - icinga group, default 'nagios'.
node['icinga']['plugin_dir'] - location where nagios plugins go,
default '/usr/lib/nagios/plugins'.

client

The following attributes are used for the client NRPE checks for warning and critical levels.

node['icinga']['client']['install_method'] - whether to install from package or source. Default chosen by platform based on known packages available for Nagios 3: debian/ubuntu 'package', redhat/centos/fedora/scientific: source
node['icinga']['plugins']['url'] - url to retrieve the plugins source
node['icinga']['plugins']['version'] - version of the plugins
node['icinga']['plugins']['checksum'] - checksum of the plugins source tarball
node['icinga']['nrpe']['home'] - home directory of nrpe, default /usr/lib/icinga
node['icinga']['nrpe']['conf_dir'] - location of the nrpe configuration, default /etc/icinga
node['icinga']['nrpe']['url'] - url to retrieve nrpe source
node['icinga']['nrpe']['version'] - version of nrpe to download
node['icinga']['nrpe']['checksum'] - checksum of the nrpe source tarball
node['icinga']['checks']['memory']['critical'] - threshold of critical memory usage, default 150
node['icinga']['checks']['memory']['warning'] - threshold of warning memory usage, default 250
node['icinga']['checks']['load']['critical'] - threshold of critical load average, default 30,20,10
node['icinga']['checks']['load']['warning'] - threshold of warning load average, default 15,10,5
node['icinga']['checks']['smtp_host'] - default relayhost to check for connectivity. Default is an empty string, set via an attribute in a role.
node['icinga']['server_role'] - the role that the icinga server will have in its run list that the clients can search for.
node['icinga']['multi_environment_monitoring'] - Allow Nagios servers in any Chef environment to monitor NRPE

server

Default directory locations are based on FHS. Change to suit your preferences.

node['icinga']['server']['install_method'] - whether to install from package or source. Default chosen by platform based on known packages available for Nagios 3: debian/ubuntu 'package', redhat/centos/fedora/scientific: source
node['icinga']['server']['service_name'] - name of the service used for icinga, default chosen by platform, debian/ubuntu "icinga", redhat family "icinga", all others, "icinga"
node['icinga']['server']['web_server'] - web server to use. supports apache or nginx, default "nginx"
node['icinga']['server']['nginx_dispatch'] - nginx dispatch method. support cgi or php, default "cgi"
node['icinga']['server']['stop_apache'] - stop apache service if using nginx, default false
node['icinga']['home'] - icinga main home directory, default "/usr/lib/icinga"
node['icinga']['conf_dir'] - location where main icinga config lives, default "/etc/icinga"
node['icinga']['config_dir'] - location where included configuration files live, default "/etc/icinga/conf.d"
node['icinga']['log_dir'] - location of nagios logs, default "/var/log/icinga"
node['icinga']['cache_dir'] - location of cached data, default "/var/cache/icinga"
node['icinga']['state_dir'] - nagios runtime state information, default "/var/lib/icinga"
node['icinga']['run_dir'] - where pidfiles are stored, default "/var/run/icinga"
node['icinga']['docroot'] - icinga webui docroot, default "/usr/share/icinga/htdocs"
node['icinga']['enable_ssl] - boolean for whether icinga web server should be https, default false
node['icinga']['http_port'] - port that the apache server should listen on, determined whether ssl is enabled (443 if so, otherwise 80)
node['icinga']['server_name'] - common name to use in a server cert, default "icinga"
node['icinga']['ssl_req'] - info to use in a cert, default /C=US/ST=Several/L=Locality/O=Example/OU=Operations/CN=#{node['icinga']['server_name']}/emailAddress=ops@#{node['icinga']['server_name']}
node['icinga']['notifications_enabled'] - set to 1 to enable notification.
node['icinga']['check_external_commands']
node['icinga']['default_contact_groups']
node['icinga']['sysadmin_email'] - default notification email.
node['icinga']['sysadmin_sms_email'] - default notification sms.
node['icinga']['server_auth_method'] - authentication with the server can be done with openid (using apache2::mod_auth_openid), or htauth (basic). The default is openid, any other value will use htauth (basic).
node['icinga']['templates']
node['icinga']['interval_length'] - minimum interval.
node['icinga']['default_host']['check_interval']
node['icinga']['default_host']['retry_interval']
node['icinga']['default_host']['max_check_attempts']
node['icinga']['default_host']['notification_interval']
node['icinga']['default_service']['check_interval']
node['icinga']['default_service']['retry_interval']
node['icinga']['default_service']['max_check_attempts']
node['icinga']['default_service']['notification_interval']

Recipes

default

Includes the icinga::client recipe.

client

Includes the correct client installation recipe based on platform, either icinga::client_package or icinga::client_source.

The client recipe searches for servers allowed to connect via NRPE that have a role named in the node['icinga']['server_role'] attribute. The recipe will also install the required packages and start the NRPE service. A custom plugin for checking memory is also added.

Searches are confined to the node's chef_environment.

Client commands for NRPE can be modified by editing the nrpe.cfg.erb template.

client_package

Installs the Nagios client libraries from packages. Default for Debian / Ubuntu systems.

client_source

Installs the Nagios client libraries from source. Default for Red Hat / CentOS / Fedora systems as native packages of Nagios 3 are not available in the default repositories.

server

Includes the correct client installation recipe based on platform, either icinga::server_package or icinga::server_source.

The server recipe sets up Apache as the web front end. The icinga::client recipe is also included. This recipe also does a number of searches to dynamically build the hostgroups to monitor, hosts that belong to them and admins to notify of events/alerts.

Searches are confined to the node's chef_environment.

The recipe does the following:

Searches for members of the sysadmins group by searching through 'users' data bag and adds them to a list for notification/contacts.
Search all nodes for a role matching the app_environment.
Search all available roles and build a list which will be the Nagios hostgroups.
Search for all nodes of each role and add the hostnames to the hostgroups.
Installs various packages required for the server.
Sets up some configuration directories.
Moves the package-installed Nagios configuration to a 'dist' directory.
Disables the 000-default VirtualHost present on Debian/Ubuntu Apache2 package installations.
Enables the Nagios web front end configuration.
Sets up the configuration templates for services, contacts, hostgroups and hosts.

NOTE: You will probably need to change the services.cfg.erb template for your environment.

To add custom commands for service checks, these can be done on a per-role basis by editing the 'services.cfg.erb' template. This template has some pre-configured checks that use role names used in an example infrastructure. Here's a brief description:

monitoring - check_smtp (e.g., postfix relayhost) w/ NRPE and tcp port 514 (e.g., rsyslog)
load_balancer - check_nginx with NRPE.
appserver - check_unicorn with NRPE, e.g. a Rails application using Unicorn.
database_master - check_mysql_server with NRPE for a MySQL database master.

server_package

Installs the Nagios server libraries from packages. Default for Debian / Ubuntu systems.

server_source

Installs the Nagios server libraries from source. Default for Red Hat / CentOS / Fedora systems as native packages of Nagios 3 are not available in the default repositories.

pagerduty

Installs and configures pagerduty plugin for icinga. You need to set a node['icinga']['pagerduty_key'] attribute on your server for this to work. This can be set through environments so that you can use different API keys for servers in production vs staging for instance.

This recipe was written based on the Nagios Integration Guide from PagerDuty which explains how to get an API key for your icinga server.

email notifications

You need to set default['icinga']['notifications_enabled'] = 1 attribute on your icinga server to enable email notifications.

For email notifications to work an appropriate mail program package and local MTA need to be installed so that /usr/bin/mail or /bin/mail is available on the system.

Example:

Include postfix cookbook to be installed on your icinga server node.

Add override_attributes to your monitoring role:

% cat roles/monitoring.rb

name "monitoring"
description "Monitoring Server"
run_list(
  "recipe[icinga::server]",
  "recipe[postfix]"
)

override_attributes(
  "icinga" => { "notifications_enabled" => "1" },
  "postfix" => { "myhostname":"your_hostname", "mydomain":"example.com" }
)

default_attributes(
  "icinga" => { "server_auth_method" => "htauth" }
)

% knife role from file monitoring.rb

Data Bags

Users

Create a users data bag that will contain the users that will be able to log into the Nagios webui. Each user can use htauth with a specified password, or an openid. Users that should be able to log in should be in the sysadmin group. Example user data bag item:

{
  "id": "nagiosadmin",
  "groups": "sysadmin",
  "htpasswd": "hashed_htpassword",
  "openid": "http://nagiosadmin.myopenid.com/",
  "icinga": {
    "pager": "nagiosadmin_pager@example.com",
    "email": "nagiosadmin@example.com"
  }
}

When using server_auth_method 'openid', use the openid in the data bag item. Any other value for this attribute (e.g., "htauth", "htpasswd", etc) will use the htpasswd value as the password in /etc/icinga/htpasswd.users.

The openid must have the http:// and trailing /. The htpasswd must be the hashed value. Get this value with htpasswd:

% htpasswd -n -s nagiosadmin
New password:
Re-type new password:
nagiosadmin:{SHA}oCagzV4lMZyS7jl2Z0WlmLxEkt4=

For example use the {SHA}oCagzV4lMZyS7jl2Z0WlmLxEkt4= value in the data bag.

Services

Create a icinga_services data bag that will contain definitions for services to be monitored. This allows you to add monitoring rules without mucking about in the services and commands templates. Each service will be named based on the id of the data bag and the command will be named withe the same id prepended with "check_". Just make sure the id in your data bag doesn't conflict with a service or command already defined in the templates.

Here's an example of a service check for sshd that you could apply to all hostgroups:

{
"id": "ssh",
"hostgroup_name": "all",
"command_line": "$USER1$/check_ssh $HOSTADDRESS$"
}

Search Defined Hostgroups

Create a icinga_hostgroups data bag that will contain definitions for Nagios hostgroups populated via search. These data bags include a Chef node search query that will populate the Nagios hostgroup with nodes based on the search.

Here's an example to find all HP hardware systems for an "hp_systems" hostgroup:

	{
	"search_query": "dmi_system_manufacturer:HP",
	"hostgroup_name": "hp_systems",
	"id": "hp_systems"
	}

Roles

Create a role to use for the monitoring server. The role name should match the value of the attribute "icinga[:server_role]". By default, this is 'monitoring'. For example:

% cat roles/monitoring.rb
name "monitoring"
description "Monitoring server"
run_list(
  "recipe[icinga::server]"
)

default_attributes(
  "icinga" => {
    "server_auth_method" => "htauth"
  }
)

% knife role from file monitoring.rb

Definitions

icinga_conf

This definition is used to drop in a configuration file in the base Nagios configuration directory's conf.d. This can be used for customized configurations for various services.

Libraries

default

The library included with the cookbook provides some helper methods used in templates.

icinga_boolean
icinga_interval - calculates interval based on interval length and a given number of seconds.
icinga_attr - retrieves a icinga attribute from the node.

Resources/Providers

The nrpecheck LWRP provides an easy way to add and remove NRPE checks from within a cookbook.

Actions

:add: creates a NRPE configuration file and restart the NRPE process. Default action.
:remove: removes the configuration file and restart the NRPE process

Attribute Parameters

command_name: name attribute. The name of the check. You'll need to reference this in your commands.cfg template
warning_condition: String that you will pass to the command with the -w flag
critical_condition: String that you will pass to the command with the -c flag
command: The actual command to execute (including the path). If this is not specified, this will use node['icinga']['plugin_dir']/command_name as the path to the command.
parameters: Any additional parameters you wish to pass to the plugin.

Examples

# Use LWRP to define check_load
icinga_nrpecheck "check_load" do
  command "#{node['icinga']['plugin_dir']}/check_load"
  warning_condition node['icinga']['checks']['load']['warning']
  critical_condition node['icinga']['checks']['load']['critical']
  action :add
end

# Remove the check_load definition
icinga_nrpecheck "check_load" do
  action :remove
end

Usage

See below under Environments for how to set up Chef 0.10 environment for use with this cookbook.

For a Nagios server, create a role named 'monitoring', and add the following recipe to the run_list:

recipe[icinga::server]

This will allow client nodes to search for the server by this role and add its IP address to the allowed list for NRPE.

To install Nagios and NRPE on a client node:

include_recipe "icinga::client"

This is a fairly complicated cookbook. For a walkthrough and example usage please see Opscode's Nagios Quick Start.

Environments

The searches used are confined to the node's chef_environment. If you do not use any environments (Chef 0.10+ feature) the _default environment is used, which is applied to all nodes in the Chef Server that are not in another defined role. To use environments, create them as files in your chef-repo, then upload them to the Chef Server.

% cat environments/production.rb
name "production"
description "Systems in the Production Environment"

% knife environment from file production.rb

License and Author

Author:: Joshua Sierles joshua@37signals.com Author:: Nathan Haneysmith nathan@opscode.com Author:: Joshua Timberman joshua@opscode.com Author:: Seth Chisamore schisamo@opscode.com Author:: Tim Smith tim.smith@webtrends.com

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Name		Name	Last commit message	Last commit date
Latest commit History 172 Commits
attributes		attributes
definitions		definitions
files/default		files/default
libraries		libraries
providers		providers
recipes		recipes
resources		resources
templates/default		templates/default
test/kitchen		test/kitchen
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING		CONTRIBUTING
Gemfile		Gemfile
LICENSE		LICENSE
README.md		README.md
metadata.rb		metadata.rb

License

OpenSourceAlchemist/icinga

Folders and files

Latest commit

History

Repository files navigation

Description

Requirements

Chef

Platform

Cookbooks

Attributes

default

client

server

Recipes

default

client

client_package

client_source

server

server_package

server_source

pagerduty

email notifications

Data Bags

Users

Services

Search Defined Hostgroups

Roles

Definitions

icinga_conf

Libraries

default

Resources/Providers

Actions

Attribute Parameters

Examples

Usage

Environments

License and Author

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages