Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable keystore for autodiscover static configuration #16306

Merged
merged 23 commits into from
Apr 29, 2020

Conversation

ChrsMark
Copy link
Member

@ChrsMark ChrsMark commented Feb 13, 2020

Close #12597.

What this PR does

This PR makes use of keystore for containers/pods that are autodiscovered with static configurations only. Hint based configuration should not have access to the keystore for security reasons.

How to test it manually

  1. Create the keystore to store password for REDIS:
./metricbeat keystore create                                                                                 
Created metricbeat keystore
./metricbeat keystore add REDIS_PASSWORD                                                                      
Enter value for REDIS_PASSWORD: 
Successfully updated the keystore
  1. Enable autodiscover with static template:
metricbeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: redis
          config:
            - module: redis
              metricsets: ["info", "keyspace"]
              hosts: ["${data.host}:6379"]
              password: ${REDIS_PASSWORD}
  1. Start metricbeat
  2. Start a password redis server in a container to be autodiscovred:
docker run -p 6379:6379 \
--entrypoint "redis-server" redis --appendonly yes --requirepass 'passpass'

Check that REDIS metrics are successfully collected.
Try to start REDIS with a different password so as to make Metricbeat fail.

Now check that hints based autodiscovered containers have not access to the keystore:

  1. Configure hint's based autodiscover:
metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
  1. Check that hints have no access to the keystore:
docker run -p 6379:6379 \
-l co.elastic.metrics/module=redis \
-l co.elastic.metrics/metricsets=info \
-l co.elastic.metrics/password='${REDIS_PASSWORD}' \
-l co.elastic.metrics/hosts='${data.host}:6379' \
--entrypoint "redis-server" redis --appendonly yes --requirepass 'passpass'

You should see Metribeat failing to access REDIS since the password is not retrievable.

cc: @exekias

Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark added containers Related to containers use case Team:Integrations Label for the Integrations team autodiscovery Team:Platforms Label for the Integrations - Platforms team labels Feb 13, 2020
@ChrsMark ChrsMark requested a review from a team February 13, 2020 14:27
@ChrsMark ChrsMark self-assigned this Feb 13, 2020
@@ -34,6 +34,8 @@ var (
ErrKeyDoesntExists = errors.New("cannot retrieve the key")
)

var CentralKeystore Keystore
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a hack. We should avoid global variables as they can cause issues when reusing code, among many other problems (I'm ok with this for testing the approach, but we need to find another solution before merging).

I think the Beat object is exposing the keystore through beat.Keystore() method. Probably you can get it somewhere and pass it to autodiscover.

Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
Copy link
Member Author

@exekias tried to pass the Keystore from instance.Beat to beat.Beat and then grab it in newMetricbeat/filebeat.Run and set it to autodiscover.Registry 😅 .

Not sure if this is better than the previous one but not sure if there is other obvious way so far. Feel free to comment/propose on it 🙂 .

Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark marked this pull request as ready for review April 24, 2020 10:53
@ChrsMark ChrsMark requested a review from a team as a code owner April 24, 2020 10:53
@ChrsMark ChrsMark requested review from a team April 24, 2020 10:55
Copy link
Contributor

@exekias exekias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for looking into this, the change is looking really clean IMO!
Would it make sense to update docs to explicitly explaining the relation between autodiscover and keystore? Probably we want to explain that keystore can be used from local settings but not from hints.

Will need a changelog

Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
Copy link
Member Author

@exekias thanks for the suggestions! So far I changed the code so as to pass keystore to NewAutodiscover and then to each autodiscover provider (think this makes things clener and more future proof). Currently only docker and kubernetes providers utilise the keystore by adding it to each event. Happy to rethink of it either now or in a next upcoming PR I will prepare in order to introduce a new keystore backend to retrieve k8s secrets.

ps: I'm adding the docs section in a following commit.

@ChrsMark
Copy link
Member Author

ChrsMark commented Apr 27, 2020

@exekias tests and docs added, however CI is still failing at https://travis-ci.org/github/elastic/beats/jobs/680132395#L1166 but not sure if this is related to these changes. I was able to reproduce the failure locally with master, running make -C generator/_templates/metricbeat test test-package.

@ChrsMark
Copy link
Member Author

ChrsMark commented Apr 27, 2020

Generator tests failing on master: #18014

fix PR: #18020

@ChrsMark ChrsMark added test-plan Add this PR to be manual test plan [zube]: In Review review labels Apr 28, 2020
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark requested a review from exekias April 28, 2020 11:20
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark added v7.8.0 needs_backport PR is waiting to be backported to other branches. labels Apr 28, 2020
@ChrsMark
Copy link
Member Author

jenkins, test this again please

@elasticmachine
Copy link
Collaborator

💔 Build Failed

Pipeline View Test View Changes Artifacts preview stats

Expand to view the summary

Build stats

Test stats 🧪

Test Results
Failed 1
Passed 7026
Skipped 1237
Total 8264

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat Windows / test_clean_removed_with_clean_inactive – test_registrar.Test
    • Status: FAILED
    • Age: 1
    • Duration: 0.575
    • Error Details:
      -------------------- >> begin captured stdout << ---------------------
      registry size: 2
      registry size after remove: 0

--------------------- >> end captured stdout << ----------------------

Steps errors

Expand to view the steps failures

  • Name: Mage update build test

    • Description: mage update build test

    • Result: FAILURE

    • Duration: 7 min 57 sec<

    • Start Time: 2020-04-29T08:37:30.140+0000

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Result: FAILURE

    • Duration: 9 min 22 sec<

    • Start Time: 2020-04-29T08:40:59.977+0000

  • Name: Make -C generator/_templates/metricbeat test

    • Description: make -C generator/_templates/metricbeat test

    • Result: FAILURE

    • Duration: 2 min 24 sec<

    • Start Time: 2020-04-29T08:37:30.581+0000

Log output

Expand to view the last 100 lines of log output

[2020-04-29T09:14:54.700Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-29T09:15:13.219Z] 
Creating appsearch_7918a246f6f8_appsearch_1      ... done
.Stopping appsearch_7918a246f6f8_appsearch_1      ... 
[2020-04-29T09:15:13.219Z] Stopping appsearch_7918a246f6f8_elasticsearch_1  ... 
[2020-04-29T09:15:14.161Z] 
Stopping appsearch_7918a246f6f8_appsearch_1      ... done

Stopping appsearch_7918a246f6f8_elasticsearch_1  ... done
Removing appsearch_7918a246f6f8_appsearch_1      ... 
[2020-04-29T09:15:14.161Z] Removing appsearch_7918a246f6f8_elasticsearch_1  ... 
[2020-04-29T09:15:15.713Z] 
Removing appsearch_7918a246f6f8_elasticsearch_1  ... done

Removing appsearch_7918a246f6f8_appsearch_1      ... done
Creating cockroachdb_7918a246f6f8_cockroachdb_1  ... 
[2020-04-29T09:15:25.240Z] 
Creating cockroachdb_7918a246f6f8_cockroachdb_1  ... done
.Stopping cockroachdb_7918a246f6f8_cockroachdb_1  ... 
[2020-04-29T09:15:25.240Z] 
Stopping cockroachdb_7918a246f6f8_cockroachdb_1  ... done
Removing cockroachdb_7918a246f6f8_cockroachdb_1  ... 
[2020-04-29T09:15:25.240Z] 
Removing cockroachdb_7918a246f6f8_cockroachdb_1  ... done
Creating coredns_7918a246f6f8_coredns_1          ... 
[2020-04-29T09:15:34.768Z] 
Creating coredns_7918a246f6f8_coredns_1          ... done
.Stopping coredns_7918a246f6f8_coredns_1          ... 
[2020-04-29T09:15:34.768Z] 
Stopping coredns_7918a246f6f8_coredns_1          ... done
Removing coredns_7918a246f6f8_coredns_1          ... 
[2020-04-29T09:15:34.768Z] 
Removing coredns_7918a246f6f8_coredns_1          ... done
Creating ibmmq_7918a246f6f8_ibmmq_1              ... 
[2020-04-29T09:15:44.091Z] >> python test: Unit Testing
[2020-04-29T09:15:58.130Z] 
Creating ibmmq_7918a246f6f8_ibmmq_1              ... done
.Stopping ibmmq_7918a246f6f8_ibmmq_1              ... 
[2020-04-29T09:15:58.130Z] 
Stopping ibmmq_7918a246f6f8_ibmmq_1              ... done
Removing ibmmq_7918a246f6f8_ibmmq_1              ... 
[2020-04-29T09:15:58.391Z] 
Removing ibmmq_7918a246f6f8_ibmmq_1              ... done
Creating mssql_7918a246f6f8_mssql_1              ... 
[2020-04-29T09:16:14.954Z] 
Creating mssql_7918a246f6f8_mssql_1              ... done
..Stopping mssql_7918a246f6f8_mssql_1              ... 
[2020-04-29T09:16:27.220Z] 
Stopping mssql_7918a246f6f8_mssql_1              ... done
Removing mssql_7918a246f6f8_mssql_1              ... 
[2020-04-29T09:16:29.139Z] 
Removing mssql_7918a246f6f8_mssql_1              ... done
Creating openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... 
[2020-04-29T09:16:35.389Z] 
Creating openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... done
.Stopping openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... 
[2020-04-29T09:16:35.389Z] 
Stopping openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... done
Removing openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... 
[2020-04-29T09:16:35.966Z] 
Removing openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... done
Creating redisenterprise_688c1f44601b6582_redisenterprise_1   ... 
[2020-04-29T09:16:51.818Z] WARNING: You are using pip version 19.2.3, however version 20.1 is available.
[2020-04-29T09:16:51.818Z] You should consider upgrading via the 'python -m pip install --upgrade pip' command.
[2020-04-29T09:16:51.818Z] S.S.SSSS
[2020-04-29T09:16:51.818Z] [success] 80.82% test_file_integrity.Test.test_non_recursive: 1.5864s
[2020-04-29T09:16:51.818Z] [success] 19.18% test_base.Test.test_start_stop: 0.3766s
[2020-04-29T09:16:51.818Z] ----------------------------------------------------------------------
[2020-04-29T09:16:51.818Z] Ran 8 tests in 1.966s
[2020-04-29T09:16:51.818Z] 
[2020-04-29T09:16:51.818Z] OK (SKIP=6)
[2020-04-29T09:16:51.818Z] >> python test: Unit Testing Complete
[2020-04-29T09:16:51.895Z] Recording test results
[2020-04-29T09:16:57.023Z] Archiving artifacts
[2020-04-29T09:18:45.385Z] 
Creating redisenterprise_688c1f44601b6582_redisenterprise_1   ... done
..Stopping redisenterprise_688c1f44601b6582_redisenterprise_1   ... 
[2020-04-29T09:18:50.704Z] 
Stopping redisenterprise_688c1f44601b6582_redisenterprise_1   ... done
Removing redisenterprise_688c1f44601b6582_redisenterprise_1   ... 
[2020-04-29T09:18:51.280Z] 
Removing redisenterprise_688c1f44601b6582_redisenterprise_1   ... done
Creating sql_7918a246f6f8_mysql_1                             ... 
[2020-04-29T09:19:07.602Z] 
Creating sql_7918a246f6f8_mysql_1                             ... done
.Stopping sql_7918a246f6f8_mysql_1                             ... 
[2020-04-29T09:19:07.864Z] 
Stopping sql_7918a246f6f8_mysql_1                             ... done
Removing sql_7918a246f6f8_mysql_1                             ... 
[2020-04-29T09:19:08.696Z] 
Removing sql_7918a246f6f8_mysql_1                             ... done
Creating stan_7918a246f6f8_stan_1                             ... 
[2020-04-29T09:19:19.171Z] 
Creating stan_7918a246f6f8_stan_1                             ... done
...Stopping stan_7918a246f6f8_stan_1                             ... 
[2020-04-29T09:19:31.439Z] 
Stopping stan_7918a246f6f8_stan_1                             ... done
Removing stan_7918a246f6f8_stan_1                             ... 
[2020-04-29T09:19:35.653Z] 
Removing stan_7918a246f6f8_stan_1                             ... done
.
[2020-04-29T09:19:35.654Z] [success] 67.23% test_xpack_base.Test.test_dashboards: 89.1094s
[2020-04-29T09:19:35.654Z] [success] 4.36% test_activemq.ActiveMqTest_0.test_broker_metrics_collected: 5.7756s
[2020-04-29T09:19:35.654Z] [success] 4.24% test_statsd.Test.test_server: 5.6251s
[2020-04-29T09:19:35.654Z] [success] 3.36% test_cockroachdb.Test.test_status: 4.4529s
[2020-04-29T09:19:35.654Z] [success] 2.11% test_xpack_base.Test.test_migration: 2.7922s
[2020-04-29T09:19:35.654Z] [success] 2.03% test_sql.Test.test_query: 2.6844s
[2020-04-29T09:19:35.654Z] [success] 1.74% test_openmetrics.Test.test_openmetrics: 2.3030s
[2020-04-29T09:19:35.654Z] [success] 1.50% test_appsearch.Test.test_stats: 1.9895s
[2020-04-29T09:19:35.654Z] [success] 1.40% test_xpack_base.Test.test_template: 1.8611s
[2020-04-29T09:19:35.654Z] [success] 1.20% test_redisenterprise.Test_0.test_metricset_1_proxy: 1.5851s
[2020-04-29T09:19:35.654Z] [success] 1.15% test_ibmmq.Test.test_qmgr: 1.5261s
[2020-04-29T09:19:35.654Z] [success] 1.10% test_stan.TestStan.test_metricset_2_subscriptions: 1.4601s
[2020-04-29T09:19:35.654Z] [success] 1.03% test_redisenterprise.Test_0.test_metricset_0_node: 1.3660s
[2020-04-29T09:19:35.654Z] [success] 1.02% test_stan.TestStan.test_metricset_1_channels: 1.3578s
[2020-04-29T09:19:35.654Z] [success] 0.94% test_stan.TestStan.test_metricset_0_stats: 1.2433s
[2020-04-29T09:19:35.654Z] [success] 0.78% test_activemq.ActiveMqTest_0.test_queue_metrics_collected: 1.0279s
[2020-04-29T09:19:35.654Z] [success] 0.72% test_mssql.Test.test_performance: 0.9547s
[2020-04-29T09:19:35.654Z] [success] 0.71% test_activemq.ActiveMqTest_1.test_queue_metrics_collected: 0.9377s
[2020-04-29T09:19:35.654Z] [success] 0.66% test_coredns.Test.test_stats: 0.8706s
[2020-04-29T09:19:35.654Z] [success] 0.58% test_mssql.Test.test_status: 0.7734s
[2020-04-29T09:19:35.654Z] [success] 0.57% test_activemq.ActiveMqTest_0.test_topic_metrics_collected: 0.7534s
[2020-04-29T09:19:35.654Z] [success] 0.54% test_activemq.ActiveMqTest_1.test_broker_metrics_collected: 0.7132s
[2020-04-29T09:19:35.654Z] [success] 0.54% test_activemq.ActiveMqTest_1.test_topic_metrics_collected: 0.7098s
[2020-04-29T09:19:35.654Z] [success] 0.51% test_xpack_base.Test.test_start_stop: 0.6766s
[2020-04-29T09:19:35.654Z] ----------------------------------------------------------------------
[2020-04-29T09:19:35.654Z] Ran 24 tests in 663.839s
[2020-04-29T09:19:35.654Z] 
[2020-04-29T09:19:35.654Z] OK
[2020-04-29T09:19:35.654Z] >> python test: Integration Testing Complete
[2020-04-29T09:19:35.921Z] >> Stopping Docker test environment...
[2020-04-29T09:19:38.535Z] Recording test results
[2020-04-29T09:19:42.753Z] Archiving artifacts
[2020-04-29T09:19:43.809Z] + curl -sSLo codecov https://codecov.io/bash
[2020-04-29T09:19:44.382Z] + FILE=auditbeat/build/coverage/full.cov
[2020-04-29T09:19:44.382Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-04-29T09:19:44.382Z] + FILE=filebeat/build/coverage/full.cov
[2020-04-29T09:19:44.382Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-04-29T09:19:44.382Z] + FILE=heartbeat/build/coverage/full.cov
[2020-04-29T09:19:44.382Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-04-29T09:19:44.382Z] + FILE=libbeat/build/coverage/full.cov
[2020-04-29T09:19:44.382Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-04-29T09:19:44.382Z] + FILE=metricbeat/build/coverage/full.cov
[2020-04-29T09:19:44.382Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-04-29T09:19:44.382Z] + FILE=packetbeat/build/coverage/full.cov
[2020-04-29T09:19:44.382Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-04-29T09:19:44.382Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-04-29T09:19:44.382Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-04-29T09:19:44.382Z] + FILE=journalbeat/build/coverage/full.cov
[2020-04-29T09:19:44.382Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-29T09:19:46.946Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-16306
[2020-04-29T09:19:47.180Z] [INFO] getVaultSecret: Getting secrets
[2020-04-29T09:19:47.243Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-04-29T09:19:47.872Z] + chmod 755 generate-build-data.sh
[2020-04-29T09:19:47.873Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-16306/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-16306/runs/23 FAILURE 4275865
[2020-04-29T09:19:48.423Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-16306/runs/23/steps/?limit=10000 -o steps-info.json
[2020-04-29T09:19:48.975Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-16306/runs/23/tests/?status=FAILED -o tests-errors.json
[2020-04-29T09:19:49.886Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-16306/runs/23/log/ -o pipeline-log.txt

@ChrsMark
Copy link
Member Author

I will go on and merge this, since Travis succeeded except for the Generator's tests.

@ChrsMark ChrsMark merged commit c1160e3 into elastic:master Apr 29, 2020
ChrsMark added a commit to ChrsMark/beats that referenced this pull request Apr 29, 2020
@ChrsMark ChrsMark removed the needs_backport PR is waiting to be backported to other branches. label Apr 29, 2020
@ChrsMark ChrsMark mentioned this pull request Apr 29, 2020
@andresrc andresrc added the test-plan-added This PR has been added to the test plan label May 3, 2020
ChrsMark added a commit that referenced this pull request May 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
autodiscovery containers Related to containers use case review Team:Integrations Label for the Integrations team Team:Platforms Label for the Integrations - Platforms team test-plan Add this PR to be manual test plan test-plan-added This PR has been added to the test plan v7.8.0
Projects
None yet
4 participants