System test for logstash module, xpack code path

[ycombinator]

Similar to #12506 but for the logstash module.

This PR introduces a system test to exercise the xpack code path (xpack.enabled: true) of the Logstash Metricbeat module.

Testing this PR

1. Build and activate the virtual env for system tests.

   cd metricbeat
   make python-env
   . build/python-env/bin/activate
   

2. Make sure fields.yml is present.

   make update
   

3. Build the test binary.

   make metricbeat.test
   

4. Run the integration tests

   INTEGRATION_TESTS=1  nosetests -v --nocapture tests/system/test_logstash.py
   

5. Assert that there are no errors.

[houndci-bot]

don't use an underscore in package name

[cachedout]

I get the following:

Step 2/4 : COPY healthcheck.sh /
 ---> Using cache
 ---> 50e24dfe04e3
Step 3/4 : ENV XPACK_MONITORING_ENABLED=FALSE
 ---> Using cache
 ---> 961e87ba94d6
Step 4/4 : HEALTHCHECK --interval=1s --retries=300 CMD sh /healthcheck.sh
 ---> Using cache
 ---> 0004b6cdc342

Successfully built 0004b6cdc342
Successfully tagged metricbeat_logstash:latest
Recreating metricbeat_logstash_1 ... done
logstash node metricset test ... {u'logstash': {u'node': {u'jvm': {u'version': u'1.8.0_201'}}}, u'service': {u'name': u'logstash', u'hostname': u'42c8d68fd285', u'version': u'7.0.0', u'address': u'localhost:32771', u'type': u'logstash', u'id': u'74910852-64df-48b2-895c-2c8e79e30fb9'}, u'process': {u'pid': 1}, u'@timestamp': u'2019-06-26T10:07:55.909Z', u'agent': {u'ephemeral_id': u'269e7fa4-fa62-4b88-aca0-8933cb812fc6', u'type': u'metricbeat', u'hostname': u'Mikes-MacBook-Pro.local', u'version': u'8.0.0', u'id': u'7c29aee5-3f10-48b3-894f-d53b5ee654b9'}, u'metricset': {u'name': u'node'}, u'host': {u'name': u'Mikes-MacBook-Pro.local'}, u'ecs': {u'version': u'1.0.1'}, u'event': {u'duration': 27335129, u'module': u'logstash', u'dataset': u'logstash.node'}}
ok
logstash node_stats metricset test ... {u'logstash': {u'node': {u'stats': {u'events': {u'filtered': 0, u'out': 0, u'in': 0}}}}, u'service': {u'name': u'logstash', u'hostname': u'42c8d68fd285', u'version': u'7.0.0', u'address': u'localhost:32771', u'type': u'logstash', u'id': u'74910852-64df-48b2-895c-2c8e79e30fb9'}, u'@timestamp': u'2019-06-26T10:07:58.397Z', u'agent': {u'ephemeral_id': u'33abeda9-ae7d-402e-a426-e1d7000b63d2', u'version': u'8.0.0', u'type': u'metricbeat', u'id': u'0806055e-3c2d-4fa7-ad29-b057837792bc', u'hostname': u'Mikes-MacBook-Pro.local'}, u'event': {u'duration': 22785723, u'module': u'logstash', u'dataset': u'logstash.node.stats'}, u'host': {u'name': u'Mikes-MacBook-Pro.local'}, u'ecs': {u'version': u'1.0.1'}, u'metricset': {u'name': u'node_stats'}}
ok
logstash-xpack module tests ... ERROR
Killing metricbeat_logstash_1    ... done

======================================================================
ERROR: logstash-xpack module tests
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/tests/system/test_logstash.py", line 73, in test_xpack
    self.wait_until(lambda: self.output_lines() > 0)
  File "/Users/mp/devel/go/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 352, in wait_until
    "Waited {} seconds.".format(max_timeout))
TimeoutError: Timeout waiting for 'cond' to be true. Waited 10 seconds.
-------------------- >> begin captured logging << --------------------
compose.config.config: DEBUG: Using configuration files: ./docker-compose.yml
docker.utils.config: DEBUG: Trying paths: ['/Users/mp/.docker/config.json', '/Users/mp/.dockercfg']
docker.utils.config: DEBUG: Found file at path: /Users/mp/.docker/config.json
docker.auth: DEBUG: Found 'auths' section
docker.auth: DEBUG: Auth data for https://index.docker.io/v1/ is absent. Client might be using a credentials store instead.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 3 tests in 36.956s

FAILED (errors=1)

[cachedout]

I had some trouble getting these tests to run without errors and put my comments in the main thread. It's possible these errors are local to me because it seems that the test suite doesn't even like it if Py3 is installed on the system and so I made some minor workarounds to try to get it to run at all.

[ycombinator]

@cachedout These tests are expected to fail at the moment. There are other PRs that need to be merged first before this PR can work. This is why I didn't put the review label on this PR yet. Unfortunately, codeowners automatically requested reviews, which makes it confusing.

[cachedout]

Ah, OK. I was not aware we were using the label for that purpose. Thanks!

[elasticmachine]

Pinging @elastic/stack-monitoring

[ycombinator]

jenkins, test this

[ycombinator]

The x-pack test failure here is legitimate. The tests run against Logstash 7.0.0 which, of course, doesn't contain the API changes needed for the logstash module (with xpack.enabled: true) to work. This means we need two changes:

1. The test in this PR needs to be skip if it's going against Logstash < 7.3.0. We have similar logic in the Kibana system test. I'll make this change in this PR itself, which should then make CI go green on this PR.

2. We also need a version check in the module code itself (when xpack.enabled: true is set). Again, we have similar logic in the Kibana module. I'll make this change in a separate PR.

[ycombinator]

The test in this PR needs to be skip if it's going against Logstash < 7.3.0.

Implemented in 7af8a99.

We also need a version check in the module code itself (when xpack.enabled: true is set).

Implemented in #12705.

[ruflin]

LGTM

I wonder what happens if Metricbeat 7.3 is run against LS 7.2. Does the user get a good error?

[ruflin]

Just found my answer: #12705

[cachedout]

In case anybody comes across this...if you are using pyenv or another system to manage multiple Python versions on your system, you must explicitly set PYTHON_EXE to point to binary used by pyenv. :)

Anyhow, I was able to get this up and working and tests pass as expected.