IndexError when multiple array elements are expected

[sebastian-schmid]

Hello Drew,

we tried to monitor our consul system with this check, which works great except the json response of consul is empty. For example consul responds with an empty json if both of two services are shut down which we want to end in a critical status.

Here is an example:

First we get two json documents and a critical status from the script:

console> ./check_http_json.py -H localhost -P 8500 -p /v1/health/service/myservice -q "(0).Node,servera" "(1).Node,serverb" -Q "(0).Status,passing" -E "(0).Status" -d
url:http://localhost:8500//v1/health/service/myservice
json:
[{u'Checks': [{u'CheckID': u'serfHealth',
               u'CreateIndex': 1589745,
               u'ModifyIndex': 1590432,
               u'Name': u'Serf Health Status',
               u'Node': u'servera',
               u'Notes': u'',
               u'Output': u'Agent alive and reachable',
               u'ServiceID': u'',
               u'ServiceName': u'',
               u'Status': u'passing'},
              {u'CheckID': u'service:b474927e-28ba-43d1-b4d8-e94bb4302adc',
               u'CreateIndex': 1591994,
               u'ModifyIndex': 1591995,
               u'Name': u"Service 'myservice' check",
               u'Node': u'servera',
               u'Notes': u'',
               u'Output': u'HTTP GET http://localhost:33000/ping: 200  Output: ',
               u'ServiceID': u'b474927e-28ba-43d1-b4d8-e94bb4302adc',
               u'ServiceName': u'myservice',
               u'Status': u'passing'}],
  u'Node': {u'Address': u'192.168.178.50',
            u'CreateIndex': 1589745,
            u'ModifyIndex': 1591995,
            u'Node': u'servera',
            u'TaggedAddresses': {u'wan': u'192.168.178.50'}},
  u'Service': {u'Address': u'',
               u'CreateIndex': 1591994,
               u'EnableTagOverride': False,
               u'ID': u'b474927e-28ba-43d1-b4d8-e94bb4302adc',
               u'ModifyIndex': 1591995,
               u'Port': 33000,
               u'Service': u'myservice',
               u'Tags': [u'Application',
                         u'develop',
                         u'b474927e-28ba-43d1-b4d8-e94bb4302adc']}},
{u'Checks': [{u'CheckID': u'serfHealth',
               u'CreateIndex': 1589964,
               u'ModifyIndex': 1590486,
               u'Name': u'Serf Health Status',
               u'Node': u'severb',
               u'Notes': u'',
               u'Output': u'Agent alive and reachable',
               u'ServiceID': u'',
               u'ServiceName': u'',
               u'Status': u'passing'},
              {u'CheckID': u'service:2aa34f51-6bdb-4ba7-91e8-2e3e7dcec66c',
               u'CreateIndex': 1591785,
               u'ModifyIndex': 1591788,
               u'Name': u"Service 'myservice' check",
               u'Node': u'severb',
               u'Notes': u'',
               u'Output': u'HTTP GET http://localhost:33000/ping: 200  Output: ',
               u'ServiceID': u'2aa34f51-6bdb-4ba7-91e8-2e3e7dcec66c',
               u'ServiceName': u'myservice',
               u'Status': u'passing'}],
  u'Node': {u'Address': u'192.168.178.50',
            u'CreateIndex': 1589964,
            u'ModifyIndex': 1591893,
            u'Node': u'severb',
            u'TaggedAddresses': {u'wan': u'192.168.178.50'}},
  u'Service': {u'Address': u'',
               u'CreateIndex': 1591785,
               u'EnableTagOverride': False,
               u'ID': u'2aa34f51-6bdb-4ba7-91e8-2e3e7dcec66c',
               u'ModifyIndex': 1591788,
               u'Port': 33000,
               u'Service': u'myservice',
               u'Tags': [u'Application',
                         u'develop',
                         u'2aa34f51-6bdb-4ba7-91e8-2e3e7dcec66c']}}]
rules:Namespace(auth=None, data=None, debug=True, headers=None, host='localhost', key_list=None, key_list_critical=['(0).Status'], key_threshold_critical=None, key_threshold_warning=None, key_value_list=['(0).Node,servera', '(1).Node,severb'], key_value_list_critical=['(0).Status,passing'], metric_list=None, path='/v1/health/service/myservice', port='8500', separator=None, ssl=False, timeout=None)
separator:.
CRITICAL: Status CRITICAL. Value for key (0).Node did not match servera. Value for key (1).Node did not match severb. Value for key (0).Status did not match passing. Key (0).Status did not exist.

Now we shut down one service so the second json document is empty, the script crashed:

console> ./check_http_json.py -H localhost -P 8500 -p /v1/health/service/myservice -q "(0).Node,servera" "(1).Node,severb" -Q "(0).Status,passing" -E "(0).Status" -d
url:http://localhost:8500//v1/health/service/myservice
json:
[{u'Checks': [{u'CheckID': u'serfHealth',
               u'CreateIndex': 1589745,
               u'ModifyIndex': 1590432,
               u'Name': u'Serf Health Status',
               u'Node': u'servera',
               u'Notes': u'',
               u'Output': u'Agent alive and reachable',
               u'ServiceID': u'',
               u'ServiceName': u'',
               u'Status': u'passing'},
              {u'CheckID': u'service:b474927e-28ba-43d1-b4d8-e94bb4302adc',
               u'CreateIndex': 1591994,
               u'ModifyIndex': 1591995,
               u'Name': u"Service 'myservice' check",
               u'Node': u'servera',
               u'Notes': u'',
               u'Output': u'HTTP GET http://localhost:33000/ping: 200  Output: ',
               u'ServiceID': u'b474927e-28ba-43d1-b4d8-e94bb4302adc',
               u'ServiceName': u'myservice',
               u'Status': u'passing'}],
  u'Node': {u'Address': u'192.168.178.50',
            u'CreateIndex': 1589745,
            u'ModifyIndex': 1591995,
            u'Node': u'servera',
            u'TaggedAddresses': {u'wan': u'192.168.178.50'}},
  u'Service': {u'Address': u'',
               u'CreateIndex': 1591994,
               u'EnableTagOverride': False,
               u'ID': u'b474927e-28ba-43d1-b4d8-e94bb4302adc',
               u'ModifyIndex': 1591995,
               u'Port': 33000,
               u'Service': u'myservice',
               u'Tags': [u'Application',
                         u'develop',
                         u'b474927e-28ba-43d1-b4d8-e94bb4302adc']}}]
rules:Namespace(auth=None, data=None, debug=True, headers=None, host='localhost', key_list=None, key_list_critical=['(0).Status'], key_threshold_critical=None, key_threshold_warning=None, key_value_list=['(0).Node,servera', '(1).Node,severb'], key_value_list_critical=['(0).Status,passing'], metric_list=None, path='/v1/health/service/myservice', port='8500', separator=None, ssl=False, timeout=None)
separator:.
Traceback (most recent call last):
  File "./check_http_json.py", line 437, in <module>
    nagios.append_warning(processor.checkWarning())
  File "./check_http_json.py", line 212, in checkWarning
    failure += self.checkEquality(self.rules.key_value_list)
  File "./check_http_json.py", line 163, in checkEquality
    if (self.helper.equals(key, v) == False):
  File "./check_http_json.py", line 99, in equals
    def equals(self, key, value): return self.exists(key) and str(self.get(key)) in value.split(':')
  File "./check_http_json.py", line 104, in exists
    def exists(self, key): return (self.get(key) != (None, 'not_found'))
  File "./check_http_json.py", line 117, in get
    return self.getSubArrayElement(key, data)
  File "./check_http_json.py", line 95, in getSubArrayElement
    return self.get(remainingKey, data[index])
IndexError: list index out of range

Looks like a bug to us.

Any ideas how to fix this?

[martialblog]

Just tested this locally and I could recreate the error. I agree, this should fail more gracefully

[martialblog]

How to recreate in the unittest:

# This should not throw an IndexError
data = '[{"Node": "there"}]'
self.check_data(rules.dash_q(['(0).Node,there', '(1).Node,missing']), data, CRITICAL_CODE)

[martialblog]

Fixed in the v2 Branch

https://github.com/drewkerrigan/nagios-http-json/tree/python3