#puppet-heka
- Overview
- Module Description - What the module does and why it is useful
- Setup - The basics of getting started with heka
- Usage - Configuration options and additional functionality
- Limitations - OS compatibility, etc.
- Development - Guide for contributing to the module
##Overview
This module installs and configures Heka, a metric/log/event router and processor.
##Module Description
This module installs Heka via the published packages and configures it by templating TOML configuration files for both Heka's global configuration and for plugins.
###Packages
This module uses the RPM packages available on Heka's Github releases page:
https://github.com/mozilla-services/heka/releases
###Init scripts and Upstart/systemd files
Because the Heka packages do not include init scripts or Upstart/systemd unit files, this module includes an Upstart file template for Red Hat/CentOS 6 and a systemd unit file for Red Hat/CentOS 7.
####systemd unit file custom parameters
You can use the $systemd_unit_file_settings
parameter in the main heka
class to specify any custom systemd unit file key/value config options you want. $systemd_unit_file_settings
is a hash and unit file K/V pairs are given as keys/values in the hash:
class { '::heka':
...
systemd_unit_file_settings => {
'RestartSec' => '30',
},
...
}
This parameter is ignored on systems other than Red Hat/CentOS 7.
Note: Keys and values must be strings or they will be ignored and not inserted into the templated unit file.
###TOML config files
####Global Heka configuration
The module templates /etc/heka.toml
. Currently, it only specifies a maxprocs
value, which you can override with the heka_max_procs
parameter:
class { '::heka':
heka_max_procs => '2',
}
To specify global configuration options the module doesn't explicitly provide parameters for, you can use the global_config_settings
parameter, which is a hash and uses the key/value pairs as the key/value pairs printed in heka.toml
:
class { '::heka':
global_config_settings => {
'poolsize' => 10000,
},
}
Heka sets any global config settings that are not specified to its own internal defaults. See Configuring hekad > Global configuration options for more info on what global configuration options are available and what Heka's internal default values are.
####TOML config files for plugins
This module has defined types and and accompanying ERB templates for a few commonly used plugins.
This module also has a more generic defined type and ERB template so you can configure any Heka plugin the module doesn't already include a defined type for.
##Setup
###Setup Requirements
This module requires the following Puppet modules:
- nanlui/staging
##Usage
###Basic usage
The following installs and configures Heka with it's default settings and manages the Heka daemon once everything is installed and configured. The maxprocs
in Heka's main config will be set to the value of the processorcount
fact. All other global config values won't be printed in the main config file unless specified as parameters. If they aren't printed in the file, Heka will use its own internal default value for each setting.
class { '::heka':}
You can optionally choose to not manage the Heka service:
class { '::heka':
manage_service => false,
}
Note: The value of this parameter is also used as the default for any plugins configured by the module and will control whether they try to notify the Heka daemon. If the Heka service is not managed by the main heka
class, make sure that you don't explicitly set any plugins to notify the daemon with their refresh_heka_service
parameter. If you do, catalog compilation will fail, as the Heka service won't be in the catalog for the plugin to send notifications to.
To specify a custom configuration option the module doesn't provide an explicit parameter for, use the global_config_settings
parameter:
class { '::heka':
global_config_settings => {
'poolsize' => 10000,
'hostname' => "\"${::fqdn}\"",
},
}
####Setting cgroup memory limits
Note: This parameter only applies to CentOS 7 systems. If used on CentOS 6 or Debian/Ubuntu systems, it will be ignored.
To set a cgroup memory limit on the Heka daemon, you can use the cgroup_memory_limit
parameter. It expects a string of numbers, followed by one of K
, M
, G
or T
, for kilobytes, megabytes, gigabytes or terabytes, respectively. Catalog compilation will fail if the value doesn't conform to this regular expression:
\d{1,}[KMGT]
Example:
class { '::heka':
...
cgroup_memory_limit = '500M',
...
}
####Purging unmanaged TOML configs
You can use the purge_unmanaged_configs
parameter to remove any TOML configs not managed by Puppet. The parameter is a boolean and defaults to true
:
class { '::heka':
...
purge_unmanaged_configs => true,
...
}
####Package versioning
To specify a particular Heka package version to install, use the package_download_url
and version
parameters:
class { '::heka':
version => '0.9.2',
package_download_url => 'https://github.com/mozilla-services/heka/releases/download/v0.9.2/heka-0_9_2-linux-amd64.rpm',
}
If you specify an older version than what is currently installed, the module will downgrade the package. However, make sure that the plugins and configuration options you're specifying with the module's plugin types are compatible with the version you are downgrading to!
####Puppet parameter and Heka data types
Booleans
Config values that are supposed to be booleans in a TOML config can be used as a boolean in your Puppet code:
::heka::plugin::input::tcpinput { 'tcpinput1':
...
refresh_heka_service => true,
...
}
Strings
Config values that are supposed to be double-quoted strings in the TOML file should be entered as single-quoted strings in your Puppet code:
::heka::plugin::input::tcpinput { 'tcpinput1':
...
address => '127.0.0.1:5565'
...
}
Note: This only applies to the included defined types already included in the module. If you use the generic heka::plugin
type or the global_config_settings
parameter of the main heka
class, you must enter your value parameters for strings as they will appear in the TOML file. This means including the double-quotes:
::heka::plugin { 'dashboard1':
type => 'DashboardOutput',
settings => {
'address' => '"0.0.0.0:4352"',
...
},
}
If you need to interpolate Facter facts, enclose the whole parameter in double-quotes and escape the internal set of dobule-quotes that need to be printed:
::heka::plugin { 'dashboard1':
type => 'DashboardOutput',
settings => {
'address' => "\"${::ipaddress_lo}:4352\"",
...
},
}
Integers
Config values that are supposed to be integers in the TOML config file can be written as integers in your Puppet code:
::heka::plugin { 'dashboard3':
type => 'DashboardOutput',
settings => {
...
'ticker_interval' => 6,
...
},
}
IP addresses and ports
IP address/port combos in Heka TOML configs are entered as double-quoted strings. You can enter them in your Puppet code as regular strings with single quotes:
::heka::plugin::input::tcpinput { 'tcpinput1':
...
address => '127.0.0.1:5565'
...
}
You can use Facter facts for IP addresses by using ${}
inside of double quotes for variable interpolation in your Puppet code:
::heka::plugin::input::tcpinput { 'tcpinput1':
...
address => "${::ipaddress_lo}:5565"
...
}
###Plugins
####Common parameters
Each plugin's defined type and the custom plugin type have the following parameters in common:
plugin_dir
: the directory the TOML plugin file will be created in; defaults to/etc/heka
.plugin_file_name
: the name of the TOML plugin file; defaults to$name.toml
.plugin_file_ensure
: the type of file resource to create; defaults tofile
.plugin_file_owner
: the owner of the TOML plugin file; defaults toroot
.plugin_file_group
: the group of the TOML plugin file; defaults toroot
.plugin_file_mode
: the mode of the TOML plugin file; defaults to0644
.plugin_file_template
: the ERB template to use to generate the TOML plugin file
####Plugin type settings
Heka allows just using the plugin's type as a name if there is only one instance of that plugin that's configured.
Plugins configured via this module, however, require explicit names. The type
setting for the plugin is already specified for the plugin types the module includes.
You'll have to specify the type if you use the heka::plugin
type to create a custom plugin config.
####Inputs
#####TcpInput
::heka::plugin::input::tcpinput { 'tcpinput1':
refresh_heka_service => true,
address => "${::ipaddress_lo}:5565"
}
#####UdpInput
::heka::plugin::input::udpinput { 'udpinput1':
refresh_heka_service => true,
address => "${::ipaddress_lo}:4484"
}
#####StatsdInput
Note: An instance of this plugin must be accompanied by an instance of the StatAccumInput
in order for Heka to start properly!
::heka::plugin::input::statsdinput { 'statsdinput1':
address => "0.0.0.0:8125"
}
####StatAccumInput
Heka documentation: StatAccumInput
Note: An instance of this plugin must be used in order for instances of the StatsdInput
plugin to work properly!
::heka::plugin::input::stataccuminput { 'stataccuminput1':
refresh_heka_service => true,
ticker_interval => 1,
emit_in_fields => true,
}
####Outputs
#####CarbonOutput
Heka documentation: CarbonOutput
::heka::plugin::output::carbonoutput { 'carbonoutput1':
address => 'graphiteserver.local:2003',
message_matcher => "Type == 'heka.statmetric'",
protocol => 'udp',
}
####Custom plugins
You can use the heka::plugin
defined type to generate TOML configs for plugins that the module doesn't already include:
::heka::plugin { 'dashboard3':
type => 'DashboardOutput',
settings => {
'address' => '"10.0.1.120:4354"',
'ticker_interval' => 6,
},
}
Note: The type
parameter is required.
Settings
settings
is a hash that will get used to generate the key/value pairs in the TOML file.
settings => {
'script_type' => '"lua"',
'filename' => '"lua_decoders/nginx_access.lua"',
},
...would generate the following in the final TOML file:
filename = "lua_decoders/nginx_access.lua"
script_type = "lua"
Subsection settings
You can use the subsetting_sections
parameter to specify any sections of subsettings a plugin might require:
subsetting_sections => {
'config' => {
'log_format' => '\'$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"\'',
'type' => '"nginx.access"',
},
}
The subsetting_sections
parameter above would generate the following TOML data:
[nginx_access_decoder.config]
log_format = '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
type = "nginx.access"
A heka::plugin
with a subsetting_sections
parameter would look like:
::heka::plugin { 'nginx_access_decoder':
refresh_heka_service => true,
type => 'SandboxDecoder',
settings => {
'script_type' => '"lua"',
'filename' => '"lua_decoders/nginx_access.lua"',
},
subsetting_sections => {
'config' => {
'log_format' => '\'$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"\'',
'type' => '"nginx.access"',
},
}
}
The rendered TOML file the above heka::plugin
generates would be:
[nginx_access_decoder]
type = "SandboxDecoder"
filename = "lua_decoders/nginx_access.lua"
script_type = "lua"
[nginx_access_decoder.config]
log_format = '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
type = "nginx.access"
##Limitations
Only Red Hat and CentOS are supported right now.
##Development
Fork this repository and create feature branches against the develop branch. Please keep one feature/change/bug fix per feature branch.
###Contribution guidelines
You are welcome to send pull requests to us - however, by doing so you agree that you are granting New Relic a non-exclusive, non-revokable, no-cost license to use the code, algorithms, patents, and ideas in that code in our products if we so choose. You also agree the code is provided as-is and you provide no warranties as to its fitness or correctness for any purpose.