Update Kafka version parsing / supported range #27720

faec · 2021-09-02T18:51:38Z

What does this PR do?

Clean up the Kafka version parsing to use Sarama's internal constants and helpers, update the documentation to the current supported version range, and add a test to detect changes to the supported range.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

…rama's built in constants

elasticmachine · 2021-09-02T18:51:40Z

Pinging @elastic/agent (Team:Agent)

elasticmachine · 2021-09-02T19:09:17Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-09-03T15:00:41.696+0000
Duration: 112 min 14 sec
Commit: 1589522

Test stats 🧪

Test	Results
Failed	0
Passed	53674
Skipped	5323
Total	58997

Trends 🧪

Log output

Expand to view the last 100 lines of log output

[2021-09-03T16:47:07.714Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-09-03T16:47:08.032Z] + gcloud auth activate-service-account --key-file ****
[2021-09-03T16:47:08.600Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-09-03T16:47:08.930Z] + gsutil -m -q cp eC1wYWNrL2ZpbGViZWF0LXdpbmRvd3MtMTAtd2luZG93cy0xMDE1ODk1MjJmZDkxNTQ3YzJiZmRjMzY2NmIyYmRhNGJjYmQxODg2MjY gs://beats-ci-temp/ci/cache/
[2021-09-03T16:47:48.793Z] tests\system\test_processors.py ........                                 [ 13%]
[2021-09-03T16:48:19.178Z] tests\system\test_reload.py ...                                          [ 14%]
[2021-09-03T16:48:19.178Z] tests\system\test_template.py s                                          [ 15%]
[2021-09-03T16:48:27.322Z] tests\system\test_timeseries.py .                                        [ 15%]
[2021-09-03T16:48:27.322Z] module\aerospike\test_aerospike.py s                                     [ 16%]
[2021-09-03T16:48:27.322Z] module\apache\test_apache.py ss                                          [ 17%]
[2021-09-03T16:48:27.593Z] module\ceph\test_ceph.py ssssssssssssssssssss                            [ 25%]
[2021-09-03T16:48:27.593Z] module\consul\test_consul.py ss                                          [ 25%]
[2021-09-03T16:48:27.593Z] module\couchbase\test_couchbase.py sss                                   [ 27%]
[2021-09-03T16:48:27.593Z] module\couchdb\test_couchdb.py s                                         [ 27%]
[2021-09-03T16:48:27.593Z] module\docker\test_docker.py sssssssss                                   [ 31%]
[2021-09-03T16:48:27.593Z] module\dropwizard\test_dropwizard.py sss                                 [ 32%]
[2021-09-03T16:48:27.593Z] module\elasticsearch\test_elasticsearch.py sssssssssss                   [ 36%]
[2021-09-03T16:48:27.593Z] module\envoyproxy\test_envoyproxy.py ss                                  [ 37%]
[2021-09-03T16:48:27.593Z] module\etcd\test_etcd.py ssssssss                                        [ 40%]
[2021-09-03T16:48:27.593Z] module\golang\test_golang.py s                                           [ 41%]
[2021-09-03T16:48:27.593Z] module\haproxy\test_haproxy.py sssssssssssssssssssssssss                 [ 51%]
[2021-09-03T16:48:37.595Z] module\http\test_http.py s.                                              [ 52%]
[2021-09-03T16:48:37.595Z] module\jolokia\test_jolokia.py ssss                                      [ 53%]
[2021-09-03T16:48:37.595Z] module\kafka\test_kafka.py ssssssssssssssssssss                          [ 61%]
[2021-09-03T16:48:37.595Z] module\kibana\test_kibana.py ss                                          [ 62%]
[2021-09-03T16:48:37.595Z] module\logstash\test_logstash.py sss                                     [ 63%]
[2021-09-03T16:48:37.595Z] module\memcached\test_memcached.py s                                     [ 64%]
[2021-09-03T16:48:37.595Z] module\mongodb\test_mongodb.py s                                         [ 64%]
[2021-09-03T16:48:37.595Z] module\munin\test_munin.py s                                             [ 65%]
[2021-09-03T16:48:37.595Z] module\mysql\test_mysql.py sssssss                                       [ 68%]
[2021-09-03T16:48:37.595Z] module\nats\test_nats.py ssssssssssssssssss                              [ 75%]
[2021-09-03T16:48:37.595Z] module\openmetrics\test_openmetrics.py s                                 [ 75%]
[2021-09-03T16:48:37.595Z] module\php_fpm\test_phpfpm.py s                                          [ 76%]
[2021-09-03T16:48:37.595Z] module\postgresql\test_postgresql.py ssssssssssssssssssss                [ 84%]
[2021-09-03T16:48:37.595Z] module\prometheus\test_prometheus.py sss                                 [ 85%]
[2021-09-03T16:48:37.595Z] module\redis\test_redis.py ssssssssssss                                  [ 90%]
[2021-09-03T16:49:02.889Z] module\system\test_system.py Cannot contact beats-ci-immutable-windows-10-1630686334251819478: java.lang.InterruptedException
[2021-09-03T16:50:35.556Z] ........s..s...s.s.                            [ 96%]
[2021-09-03T16:50:35.556Z] module\traefik\test_traefik.py s                                         [ 97%]
[2021-09-03T16:50:35.556Z] module\uwsgi\test_uwsgi.py ss                                            [ 97%]
[2021-09-03T16:50:35.556Z] module\vsphere\test_vsphere.py sss                                       [ 99%]
[2021-09-03T16:50:35.556Z] module\zookeeper\test_zookeeper.py ss                                    [100%]
[2021-09-03T16:50:35.556Z] 
[2021-09-03T16:50:35.556Z] ============================== warnings summary ===============================
[2021-09-03T16:50:35.556Z] tests\system\test_lightmodules.py:57
[2021-09-03T16:50:35.556Z]   C:\Users\jenkins\workspace\PR-27720-4-079606a8-6b84-4422-8501-27658b32b427\src\github.com\elastic\beats\metricbeat\tests\system\test_lightmodules.py:57: PytestCollectionWarning: cannot collect test class 'TestHTTPHandler' because it has a __init__ constructor (from: metricbeat/tests/system/test_lightmodules.py)
[2021-09-03T16:50:35.556Z]     class TestHTTPHandler(http.server.BaseHTTPRequestHandler):
[2021-09-03T16:50:35.556Z] 
[2021-09-03T16:50:35.556Z] -- Docs: https://docs.pytest.org/en/stable/warnings.html
[2021-09-03T16:50:35.556Z] - generated xml file: C:\Users\jenkins\workspace\PR-27720-4-079606a8-6b84-4422-8501-27658b32b427\src\github.com\elastic\beats\metricbeat\build\TEST-python-unit.xml -
[2021-09-03T16:50:35.556Z] ============================ slowest 20 durations =============================
[2021-09-03T16:50:35.556Z] 17.04s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_list
[2021-09-03T16:50:35.556Z] 16.99s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_disable
[2021-09-03T16:50:35.556Z] 16.97s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_enable
[2021-09-03T16:50:35.556Z] 12.59s call     metricbeat/tests/system/test_reload.py::Test::test_start_stop
[2021-09-03T16:50:35.556Z] 12.46s call     metricbeat/tests/system/test_config.py::ConfigTest::test_service_name
[2021-09-03T16:50:35.556Z] 11.11s call     metricbeat/module/system/test_system.py::Test::test_process
[2021-09-03T16:50:35.556Z] 10.87s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_test_error
[2021-09-03T16:50:35.556Z] 10.72s call     metricbeat/tests/system/test_processors.py::Test::test_dropevent_with_condition
[2021-09-03T16:50:35.556Z] 10.06s call     metricbeat/module/system/test_system.py::Test::test_fsstat
[2021-09-03T16:50:35.556Z] 9.88s call     metricbeat/module/system/test_system.py::Test::test_filesystem
[2021-09-03T16:50:35.556Z] 9.64s call     metricbeat/tests/system/test_reload.py::Test::test_reload
[2021-09-03T16:50:35.556Z] 9.26s call     metricbeat/tests/system/test_processors.py::Test::test_dropevent_with_complex_condition
[2021-09-03T16:50:35.556Z] 9.11s call     metricbeat/tests/system/test_processors.py::Test::test_multiple_actions
[2021-09-03T16:50:35.556Z] 9.11s call     metricbeat/tests/system/test_processors.py::Test::test_include_fields
[2021-09-03T16:50:35.556Z] 9.09s call     metricbeat/tests/system/test_processors.py::Test::test_contradictory_multiple_actions
[2021-09-03T16:50:35.556Z] 9.08s call     metricbeat/tests/system/test_processors.py::Test::test_dropfields_with_condition
[2021-09-03T16:50:35.556Z] 9.04s call     metricbeat/tests/system/test_timeseries.py::TestTimeseries::test_enable_timeseries
[2021-09-03T16:50:35.556Z] 9.03s call     metricbeat/tests/system/test_base.py::Test::test_export_index_pattern_migration
[2021-09-03T16:50:35.556Z] 8.96s call     metricbeat/tests/system/test_base.py::Test::test_export_index_pattern
[2021-09-03T16:50:35.556Z] 8.93s call     metricbeat/tests/system/test_base.py::Test::test_export_template
[2021-09-03T16:50:35.556Z] =========== 40 passed, 207 skipped, 1 warning in 405.16s (0:06:45) ============
[2021-09-03T16:50:35.556Z] >> python test: Unit Testing Complete
[2021-09-03T16:50:35.947Z] 
[2021-09-03T16:50:35.948Z] C:\Users\jenkins\workspace\PR-27720-4-079606a8-6b84-4422-8501-27658b32b427\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-09-03T16:50:38.184Z] 
[2021-09-03T16:50:38.185Z] C:\Users\jenkins\workspace\PR-27720-4-079606a8-6b84-4422-8501-27658b32b427\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-09-03T16:50:40.096Z] Copy .\metricbeat\build into build\metricbeat\build
[2021-09-03T16:50:40.096Z] Copy .\metricbeat\null\build into build\metricbeat\null\build
[2021-09-03T16:50:40.117Z] Running in C:\Users\jenkins\workspace\PR-27720-4-079606a8-6b84-4422-8501-27658b32b427\src\github.com\elastic\beats\build
[2021-09-03T16:50:40.137Z] Recording test results
[2021-09-03T16:51:40.492Z] [Checks API] No suitable checks publisher found.
[2021-09-03T16:51:40.891Z] 
[2021-09-03T16:51:40.891Z] C:\Users\jenkins\workspace\PR-27720-4-079606a8-6b84-4422-8501-27658b32b427\src\github.com\elastic\beats>go clean -modcache 
[2021-09-03T16:52:50.185Z] + gsutil --version
[2021-09-03T16:52:51.610Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-09-03T16:52:51.934Z] + gcloud auth activate-service-account --key-file ****
[2021-09-03T16:52:52.503Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-09-03T16:52:52.827Z] + gsutil -m -q cp bWV0cmljYmVhdC13aW5kb3dzLTEwLXdpbmRvd3MtMTAxNTg5NTIyZmQ5MTU0N2MyYmZkYzM2NjZiMmJkYTRiY2JkMTg4NjI2 gs://beats-ci-temp/ci/cache/
[2021-09-03T16:52:54.420Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-09-03T16:52:54.470Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-09-03T16:52:54.549Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-27720/src/github.com/elastic/beats
[2021-09-03T16:52:54.920Z] Running on worker-395930 in /var/lib/jenkins/workspace/Beats_beats_PR-27720
[2021-09-03T16:52:54.982Z] [INFO] getVaultSecret: Getting secrets
[2021-09-03T16:52:55.039Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-09-03T16:52:57.275Z] + chmod 755 generate-build-data.sh
[2021-09-03T16:52:57.275Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27720/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27720/runs/4 FAILURE 6734174
[2021-09-03T16:52:57.275Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27720/runs/4/steps/?limit=10000 -o steps-info.json
[2021-09-03T16:53:08.259Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27720/runs/4/tests/?status=FAILED -o tests-errors.json
[2021-09-03T16:53:11.400Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27720/runs/4/log/ -o pipeline-log.txt

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	53674
Skipped	5323
Total	58997

mergify · 2021-09-03T15:00:06Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kafka-version upstream/kafka-version
git merge upstream/master
git push upstream kafka-version

(cherry picked from commit f083957)

* master: (39 commits) [Heartbeat] Move JSON tests from python->go (elastic#27816) docs: simplify permissions for Dockerfile COPY (elastic#27754) Osquerybeat: Fix osquery logger plugin severy levels mapping (elastic#27789) [Filebeat] Update compatibility function to remove processor description on ES < 7.9.0 (elastic#27774) warn log entry and no validation failure when both queue_url and buck… (elastic#27612) libbeat/cmd/instance: ensure test config file has appropriate permissions (elastic#27178) [Heartbeat] Add httpcommon options to ZipURL (elastic#27699) Add a header round tripper option to httpcommon (elastic#27509) [Elastic Agent] Add validation to ensure certificate paths are absolute. (elastic#27779) Rename dashboards according to module.yml files for master (elastic#27749) Refactor vagrantfile, add scripts for provisioning with docker/kind (elastic#27726) Accept syslog dates with leading 0 (elastic#27775) [Filebeat] Add timezone config option to decode_cef and syslog input (elastic#27727) [Filebeat] Threatintel compatibility updates (elastic#27323) Add support for ephemeral containers in elastic agent dynamic provider (elastic#27707) [Filebeat] Integration tests in CI for AWS-S3 input (elastic#27491) Fix flakyness of TestFilestreamEmptyLine (elastic#27705) [Filebeat] kafka v2 using parsers (elastic#27335) Update Kafka version parsing / supported range (elastic#27720) Update Sarama to 1.29.1 (elastic#27717) ...

(cherry picked from commit f083957) Co-authored-by: Fae Charlton <fae.charlton@elastic.co>

faec added 5 commits September 2, 2021 13:05

Update to the latest sarama baseline of 1.29.1

d21917e

update changelog

67ebe95

Clean up the kafka version parsing to support current versions and sa…

04488d8

…rama's built in constants

Update the docs to list current Kafka versions

4dfb88d

Add a test to make future sarama updates smoother

c170509

faec added enhancement cleanup Team:Elastic-Agent Label for the Agent team backport-v7.16.0 Automated backport with mergify labels Sep 2, 2021

faec self-assigned this Sep 2, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 2, 2021

faec changed the title ~~Kafka version~~ Update Kafka version parsing / supported range Sep 2, 2021

faec added 2 commits September 2, 2021 15:04

Add a unit test for version parsing

32efc38

update changelog

526175d

faec requested a review from kvch September 2, 2021 19:10

kvch approved these changes Sep 3, 2021

View reviewed changes

Merge branch 'master' into kafka-version

1589522

faec merged commit f083957 into elastic:master Sep 3, 2021

faec deleted the kafka-version branch September 3, 2021 19:16

mergify bot pushed a commit that referenced this pull request Sep 3, 2021

Update Kafka version parsing / supported range (#27720)

cfd5df6

(cherry picked from commit f083957)

mergify bot mentioned this pull request Sep 3, 2021

[7.x](backport #27720) Update Kafka version parsing / supported range #27753

Merged

faec added a commit that referenced this pull request Sep 14, 2021

Update Kafka version parsing / supported range (#27720) (#27753)

fc6029f

(cherry picked from commit f083957) Co-authored-by: Fae Charlton <fae.charlton@elastic.co>

Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021

Update Kafka version parsing / supported range (elastic#27720)

7058097

rdner mentioned this pull request Mar 22, 2022

[Beats output.kafka] Support Kafka 2.8.1 in beats output.kafka.version field #30937

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Kafka version parsing / supported range #27720

Update Kafka version parsing / supported range #27720

faec commented Sep 2, 2021 •

edited

Loading

elasticmachine commented Sep 2, 2021

elasticmachine commented Sep 2, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

mergify bot commented Sep 3, 2021

Update Kafka version parsing / supported range #27720

Update Kafka version parsing / supported range #27720

Conversation

faec commented Sep 2, 2021 • edited Loading

What does this PR do?

Checklist

elasticmachine commented Sep 2, 2021

elasticmachine commented Sep 2, 2021 • edited by jenkins-beats-ci bot Loading

💔 Build Failed

Build stats

Test stats 🧪

Trends 🧪

Log output

💚 Flaky test report

Test stats 🧪

mergify bot commented Sep 3, 2021

faec commented Sep 2, 2021 •

edited

Loading

elasticmachine commented Sep 2, 2021 •

edited by jenkins-beats-ci bot

Loading