App::ElasticSearch::Utilities - Utilities for Monitoring ElasticSearch
version 8.8
This library contains utilities for unified interfaces in the scripts.
This a set of utilities to make monitoring ElasticSearch clusters much simpler.
Included are:
SEARCHING:
scripts/es-search.pl - Utility to interact with LogStash style indices from the CLI
MONITORING:
scripts/es-graphite-dynamic.pl - Perform index maintenance on daily indexes
scripts/es-status.pl - Command line utility for ES Metrics
scripts/es-storage-overview.pl - View how shards/data is aligned on your cluster
scripts/es-nodes.pl - View node information
MAINTENANCE:
scripts/es-daily-index-maintenance.pl - Perform index maintenance on daily indexes
scripts/es-alias-manager.pl - Manage index aliases automatically
scripts/es-open.pl - Open any closed indices matching a index parameters
MANAGEMENT:
scripts/es-apply-settings.pl - Apply settings to all indexes matching a pattern
scripts/es-cluster-settings.pl - Manage cluster settings
scripts/es-copy-index.pl - Copy an index from one cluster to another
scripts/es-storage-overview.pl - View how shards/data is aligned on your cluster
DEPRECATED:
scripts/es-graphite-static.pl - Send ES Metrics to Graphite or Cacti
The App::ElasticSearch::Utilities module simply serves as a wrapper around the scripts for packaging and distribution.
Takes an optional reference to an @ARGV
like array. Performs environment and
argument parsing.
Grab the value of the global value from the es-utils.yaml files.
Get the user/password combination for this host. This is called from LWP::UserAgent if it recieves a 401, so the auth condition must be satisfied.
Returns the username and password as a list.
Called from es_basic_auth to exec a program, capture the password and return it to the caller. This allows the use of password vaults and keychains.
Returns a hashref of the pattern filter used to get the indexes { string => '*', re => '.*', }
Without options, this connects to the server defined in the args. If passed an array ref, it will use that as the connection definition.
Returns true (1) if the handle is to the the cluster master, or false (0) otherwise.
Retrieve URL from ElasticSearch, returns a hash reference
First hash ref contains options, including:
uri_param Query String Parameters
index Index name
type Index type
method Default is GET
If the request is not successful, this function will throw a fatal exception. If you'd like to proceed you need to catch that error.
Returns the hash of index meta data.
Returns the hash of index meta data.
Returns a list of active indexes matching the filter criteria specified on the command line. Can handle indices named:
logstash-YYYY.MM.DD
dcid-logstash-YYYY.MM.DD
logstash-dcid-YYYY.MM.DD
logstash-YYYY.MM.DD-dcid
Makes use of --datesep to determine where the date is.
Options include:
-
state
Default is 'open', can be used to find 'closed' indexes as well.
-
check_state
Default is 1, set to 0 to disable state checks. The combination of the default with this option and the default for state means only open indices are returned.
-
check_dates
Default is 1, set to 0 to disable checking index age.
Returns the index name with the date removed.
Returns an array of the possible index base names for this index
Return the number of days old this index is.
Returns the number of replicas for a given index.
Checks if the specified index is valid
Returns a hash reference with the following data:
key_name:
type: field_data_type
# If the field is nested
nested_path: nested_path
nested_key: nested_key
Closes an index
Open an index
Deletes an index
Optimize an index to a single segment per shard
Apply a HASH of settings to an index.
Exposes GET /$index/_segments
Returns the segment data from the index in hashref:
Return the number of shards and segments in an index as a hashref
Exposes GET /$index/_stats
Returns a hashref
Exposes GET /_settings
Returns a hashref
Exposes GET /_nodes/stats
Returns a hashref
Performs flattening that's compatible with Elasticsearch's flattening.
Takes a number and returns the number as a string in docs, thousands, millions, or billions.
1_000 -> "1.00 thousand",
1_000_000 -> "1.00 million",
Takes a number and returns the number as a string in bytes, Kb, Mb, Gb, or Tb using base 1024.
1024 -> '1.00 Kb',
1048576 -> '1.00 Mb',
1073741824 -> '1.00 Gb',
Exposes Definitions grabbed by options parsing
Fetch meta-data from the local config file, i.e. ~/.es-utils.yaml
.
Format is:
---
meta:
index_name:
key: value
index_basename:
key: value
The most specific version is searched first, followed by the index stripped of
it's date, and then on through all the bases discovered with
es_index_bases()
.
This is used by the es-search.pl
utility to do lookups of the timestamp
field it needs to sort documents, i.e.:
---
meta:
logstash:
timestamp: '@timestamp'
host: es-cluster-01.int.example.com
bro:
timestamp: 'timestamp'
From App::ElasticSearch::Utilities:
--local Use localhost as the elasticsearch host
--host ElasticSearch host to connect to
--port HTTP port for your cluster
--proto Defaults to 'http', can also be 'https'
--http-username HTTP Basic Auth username
--password-exec Script to run to get the users password
--insecure Don't verify TLS certificates
--cacert Specify the TLS CA file
--capath Specify the directory with TLS CAs
--cert Specify the path to the client certificate
--key Specify the path to the client private key file
--noop Any operations other than GET are disabled, can be negated with --no-noop
--timeout Timeout to ElasticSearch, default 10
--keep-proxy Do not remove any proxy settings from %ENV
--index Index to run commands against
--base For daily indexes, reference only those starting with "logstash"
(same as --pattern logstash-* or logstash-DATE)
--pattern Use a pattern to operate on the indexes
--days If using a pattern or base, how many days back to go, default: 1
See also the "CONNECTION ARGUMENTS" and "INDEX SELECTION ARGUMENTS" sections from App::ElasticSearch::Utilities.
Some options may be specified in the /etc/es-utils.yaml, $HOME/.es-utils.yaml or $HOME/.config/es-utils/config.yaml file:
---
base: logstash
days: 7
host: esproxy.example.com
port: 80
timeout: 10
proto: https
http-username: bob
password-exec: /home/bob/bin/get-es-passwd.sh
Arguments for establishing a connection with the cluster. Unless specified otherwise, these options can all be set in the globals file.
-
local
Assume ElasticSearch is running locally, connect to localhost.
-
host
Use a different hostname or IP address to connect.
-
port
Defaults to 9200.
-
proto
Defaults to 'http', can also be 'https'.
-
http-username
If HTTP Basic Authentication is required, use this username.
See also the "HTTP Basic Authentication" section for more details
-
password-exec
If HTTP Basic Authentication is required, run this command, passing the arguments:
<command_to_run> <es_host> <es_username>
The script expects the last line to contain the password in plaintext.
-
noop
Prevents any communication to the cluster from making changes to the settings or data contained therein. In short, it prevents anything but HEAD and GET requests, except POST requests to the _search endpoint.
-
timeout
Timeout for connections and requests, defaults to 10.
-
keep-proxy
By default, HTTP proxy environment variables are stripped. Use this option to keep your proxy environment variables in tact.
-
insecure
Don't verify TLS certificates
-
cacert
Specify a file with the TLS CA certificates.
-
capath
Specify a directory containing the TLS CA certificates.
-
cert
Specify the path to the TLS client certificate file..
-
key
Specify the path to the TLS client private key file.
-
base
In an environment using monthly, weekly, daily, or hourly indexes. The base index name is everything without the date. Parsing for bases, also provides splitting and matching on segments of the index name delineated by the '-' character. If we have the following indexes:
web-dc1-YYYY.MM.DD web-dc2-YYYY.MM.DD logstash-dc1-YYYY.MM.DD logstash-dc2-YYYY.MM.DD
Valid bases would be:
web web-dc1 web-dc2 logstash logstash-dc1 logstash-dc2 dc1 dc2
Combining that with the days option can provide a way to select many indexes at once.
-
days
How many days backwards you want your operation to be relevant.
-
datesep
Default is '.' Can be set to an empty string for no separator.
-
pattern
A pattern to match the indexes. Can expand the following key words and characters:
'*' expanded to '.*' 'ANY' expanded to '.*' 'DATE' expanded to a pattern to match a date,
The indexes are compared against this pattern.
HTTP Basic Authorization is only supported when the proto
is set to https
as not to leak credentials all over.
The username is selected by going through these mechanisms until one is found:
--http-username
'http-username' in /etc/es-utils.yml or ~/.es-utils.yml
Netrc element matching the hostname of the request
CLI::Helpers prompt()
Once the username has been resolved, the following mechanisms are tried in order:
Netrc element matching the hostname of the request
Password executable defined by --password-exec
'password-exec' in /etc/es-utils.yml, ~/.es-utils.yml
CLI::Helpers prompt()
It is BAD practice to specify passwords as a command line argument, or store it in a plaintext file. There are cases where this may be necessary, but it is not recommended. The best method for securing your password is to use the password-exec option.
This option must point to an executable script. That script will be passed two arguments, the hostname and the username for the request. It expects the password printed to STDOUT as the last line of output. Here's an example password-exec setup using Apple Keychain:
#!/bin/sh
HOSTNAME=$1;
USERNAME=$2;
/usr/bin/security find-generic-password -w -a "$USERNAME" -s "$HOSTNAME"
If we save this to "$HOME/bin/get-passwd.sh" we can execute a script like this:
$ es-search.pl --http-username bob --password-exec $HOME/bin/get-passwd.sh \
--base secure-data --fields
Though it's probably best to set this in your ~/.es-utils.yml file:
---
host: secured-cluster.example.org
port: 443
proto: https
http-username: bob
password-exec: /home/bob/bin/get-passwd.sh
If all the fails to yield a password, the last resort is to use CLI::Helpers::prompt() to ask the user for their password. If the user is using version 1.1 or higher of CLI::Helpers, this call will turn off echo and readline magic for the password prompt.
This library attempts to provide scripts compatible with version 0.19 through 1.1 of ElasticSearch.
Recommended install with CPAN Minus:
cpanm App::ElasticSearch::Utilities
You can also use CPAN:
cpan App::ElasticSearch::Utilities
Or if you'd prefer to manually install:
export RELEASE=<CurrentRelease>
wget "https://github.com/reyjrar/es-utils/blob/master/releases/App-ElasticSearch-Utilities-$RELEASE.tar.gz?raw=true" -O es-utils.tgz
tar -zxvf es-utils.tgz
cd App-ElasticSearch-Utilities-$RELEASE
perl Makefile.PL
make
make install
This will take care of ensuring all the dependencies are satisfied and will install the scripts into the same directory as your Perl executable.
The tools are all wrapped in their own documentation, please see:
$UTILITY --help
$UTILITY --manual
For individual options and capabilities
Patterns are used to match an index to the aliases it should have. A few symbols are expanded into regular expressions. Those patterns are:
* expands to match any number of any characters.
DATE expands to match YYYY.MM.DD, YYYY-MM-DD, or YYYYMMDD
ANY expands to match any number of any characters.
Brad Lhotsky brad@divisionbyzero.net
- Alexey Shatlovsky alexey.shatlovsky@booking.com
- Samit Badle Samit.Badle@gmail.com
- Takumi Sakamoto takumi.saka@gmail.com
- Vitaly Shupak vitaly.shupak@deshaw.com
- Alexey Surikov ksurent@gmail.com
- Andrei Grechkin andrei.grechkin@booking.com
- Daniel Ostermeier daniel.ostermeier@gmail.com
- Jason Rojas jason.rojas@mgo.com
- Kang-min Liu gugod@gugod.org
- Lisa Hare lhare@inview.co.uk
- Markus Linnala Markus.Linnala@cybercom.com
- Matthew Feinberg mattf@intex.com
- Mohammad S Anwar mohammad.anwar@yahoo.com
The following websites have more information about this module, and may be of help to you. As always, in addition to those websites please use your favorite search engine to discover more resources.
-
MetaCPAN
A modern, open-source CPAN search engine, useful to view POD in HTML format.
-
CPAN Testers
The CPAN Testers is a network of smoke testers who run automated tests on uploaded CPAN distributions.
http://www.cpantesters.org/distro/A/App-ElasticSearch-Utilities
-
CPAN Testers Matrix
The CPAN Testers Matrix is a website that provides a visual overview of the test results for a distribution on various Perls/platforms.
http://matrix.cpantesters.org/?dist=App-ElasticSearch-Utilities
This module uses the GitHub Issue Tracker: https://github.com/reyjrar/es-utils/issues
This module's source code is available by visiting: https://github.com/reyjrar/es-utils
This software is Copyright (c) 2024 by Brad Lhotsky.
This is free software, licensed under:
The (three-clause) BSD License