Add automatic retries and exponential backoff to Filebeat httpjson input #18956

alakahakai · 2020-06-03T21:49:04Z

Add automatic retries and exponential backoff to Filebeat httpjson input. If an HTTP error is returned by the client, e.g., connection errors, or if a 500-range response code is received (except 501), then a retry is automatically invoked after a defined wait period.

go test result:

=== RUN   TestProviderCanonical
--- PASS: TestProviderCanonical (0.00s)
=== RUN   TestGetProviderIsCanonical
--- PASS: TestGetProviderIsCanonical (0.00s)
=== RUN   TestIsEnabled
--- PASS: TestIsEnabled (0.00s)
=== RUN   TestGetTokenURL
--- PASS: TestGetTokenURL (0.00s)
=== RUN   TestGetTokenURLWithAzure
--- PASS: TestGetTokenURLWithAzure (0.00s)
=== RUN   TestGetEndpointParams
--- PASS: TestGetEndpointParams (0.00s)
=== RUN   TestGetEndpointParamsWithAzure
--- PASS: TestGetEndpointParamsWithAzure (0.00s)
=== RUN   TestConfigValidationCase1
--- PASS: TestConfigValidationCase1 (0.00s)
=== RUN   TestConfigValidationCase2
--- PASS: TestConfigValidationCase2 (0.00s)
=== RUN   TestConfigValidationCase3
--- PASS: TestConfigValidationCase3 (0.00s)
=== RUN   TestConfigValidationCase4
--- PASS: TestConfigValidationCase4 (0.00s)
=== RUN   TestConfigValidationCase5
--- PASS: TestConfigValidationCase5 (0.00s)
=== RUN   TestConfigValidationCase6
--- PASS: TestConfigValidationCase6 (0.00s)
=== RUN   TestConfigValidationCase7
--- PASS: TestConfigValidationCase7 (0.00s)
=== RUN   TestConfigOauth2Validation
=== RUN   TestConfigOauth2Validation/can't_set_oauth2_and_api_key_together
=== RUN   TestConfigOauth2Validation/can_set_oauth2_and_api_key_together_if_oauth2_is_disabled
=== RUN   TestConfigOauth2Validation/can't_set_oauth2_and_authentication_scheme
=== RUN   TestConfigOauth2Validation/token_url_and_client_credentials_must_be_set
=== RUN   TestConfigOauth2Validation/must_fail_with_an_unknown_provider
=== RUN   TestConfigOauth2Validation/azure_must_have_either_tenant_id_or_token_url
=== RUN   TestConfigOauth2Validation/azure_must_have_only_one_of_token_url_and_tenant_id
=== RUN   TestConfigOauth2Validation/azure_must_have_client_credentials_set
=== RUN   TestConfigOauth2Validation/azure_config_is_valid
=== RUN   TestConfigOauth2Validation/google_can't_have_token_url_or_client_credentials_set
=== RUN   TestConfigOauth2Validation/google_must_fail_if_no_ADC_available
=== RUN   TestConfigOauth2Validation/google_must_fail_if_credentials_file_not_found
=== RUN   TestConfigOauth2Validation/google_must_fail_if_ADC_is_wrongly_set
=== RUN   TestConfigOauth2Validation/google_must_work_if_ADC_is_set_up
=== RUN   TestConfigOauth2Validation/google_must_work_if_credentials_file_is_correct
=== RUN   TestConfigOauth2Validation/google_must_work_if_jwt_file_is_correct
=== RUN   TestConfigOauth2Validation/google_must_work_if_credentials_json_is_correct
=== RUN   TestConfigOauth2Validation/google_must_fail_if_credentials_json_is_not_a_valid_JSON
=== RUN   TestConfigOauth2Validation/google_must_fail_if_the_provided_credentials_file_is_not_a_valid_JSON
--- PASS: TestConfigOauth2Validation (0.00s)
    --- PASS: TestConfigOauth2Validation/can't_set_oauth2_and_api_key_together (0.00s)
    --- PASS: TestConfigOauth2Validation/can_set_oauth2_and_api_key_together_if_oauth2_is_disabled (0.00s)
    --- PASS: TestConfigOauth2Validation/can't_set_oauth2_and_authentication_scheme (0.00s)
    --- PASS: TestConfigOauth2Validation/token_url_and_client_credentials_must_be_set (0.00s)
    --- PASS: TestConfigOauth2Validation/must_fail_with_an_unknown_provider (0.00s)
    --- PASS: TestConfigOauth2Validation/azure_must_have_either_tenant_id_or_token_url (0.00s)
    --- PASS: TestConfigOauth2Validation/azure_must_have_only_one_of_token_url_and_tenant_id (0.00s)
    --- PASS: TestConfigOauth2Validation/azure_must_have_client_credentials_set (0.00s)
    --- PASS: TestConfigOauth2Validation/azure_config_is_valid (0.00s)
    --- PASS: TestConfigOauth2Validation/google_can't_have_token_url_or_client_credentials_set (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_no_ADC_available (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_credentials_file_not_found (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_ADC_is_wrongly_set (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_work_if_ADC_is_set_up (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_work_if_credentials_file_is_correct (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_work_if_jwt_file_is_correct (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_work_if_credentials_json_is_correct (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_credentials_json_is_not_a_valid_JSON (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_the_provided_credentials_file_is_not_a_valid_JSON (0.00s)
=== RUN   TestGetNextLinkFromHeader
--- PASS: TestGetNextLinkFromHeader (0.00s)
=== RUN   TestCreateRequestInfoFromBody
--- PASS: TestCreateRequestInfoFromBody (0.00s)
=== RUN   TestGetRateLimitCase1
--- PASS: TestGetRateLimitCase1 (0.00s)
=== RUN   TestGetRateLimitCase2
--- PASS: TestGetRateLimitCase2 (0.00s)
=== RUN   TestGetRateLimitCase3
--- PASS: TestGetRateLimitCase3 (0.00s)
=== RUN   TestGET
2020-06-16T20:53:14.947-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53858"}
2020-06-16T20:53:14.947-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53858
--- PASS: TestGET (0.00s)
=== RUN   TestGetHTTPS
2020-06-16T20:53:14.948-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53860"}
2020-06-16T20:53:14.948-0700	WARN	[tls]	tlscommon/tls_config.go:83	SSL/TLS verifications disabled.
2020-06-16T20:53:14.948-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53860
--- PASS: TestGetHTTPS (0.00s)
=== RUN   TestRateLimitRetry
2020-06-16T20:53:14.949-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53862"}
2020-06-16T20:53:14.949-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53862
2020-06-16T20:53:14.949-0700	DEBUG	[httpjson]	httpjson/input.go:325	HTTP request failed	{"url": "http://127.0.0.1:53862", "http.response.status_code": 429, "http.response.body": ""}
2020-06-16T20:53:14.949-0700	DEBUG	[httpjson]	httpjson/input.go:262	Rate Limit: No need to apply rate limit.	{"url": "http://127.0.0.1:53862"}
2020-06-16T20:53:14.949-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53862
--- PASS: TestRateLimitRetry (0.00s)
=== RUN   TestErrorRetry
2020-06-16T20:53:14.950-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53865"}
2020-06-16T20:53:14.950-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53865
2020-06-16T20:53:14.950-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:600	[DEBUG] GET http://127.0.0.1:53865 (status: 506): retrying in 1s (5 left)
2020-06-16T20:53:15.951-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:600	[DEBUG] GET http://127.0.0.1:53865 (status: 508): retrying in 2s (4 left)
--- PASS: TestErrorRetry (3.01s)
=== RUN   TestArrayResponse
2020-06-16T20:53:17.955-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53900"}
2020-06-16T20:53:17.956-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53900
--- PASS: TestArrayResponse (0.00s)
=== RUN   TestPOST
2020-06-16T20:53:17.957-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53902"}
2020-06-16T20:53:17.957-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53902
--- PASS: TestPOST (0.00s)
=== RUN   TestRepeatedPOST
2020-06-16T20:53:17.958-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53904"}
2020-06-16T20:53:17.958-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53904
2020-06-16T20:53:20.963-0700	INFO	[httpjson]	httpjson/input.go:438	Process another repeated request.	{"url": "http://127.0.0.1:53904"}
2020-06-16T20:53:20.963-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53904
2020-06-16T20:53:23.962-0700	INFO	[httpjson]	httpjson/input.go:438	Process another repeated request.	{"url": "http://127.0.0.1:53904"}
2020-06-16T20:53:23.962-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53904
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:435	Context done.	{"url": "http://127.0.0.1:53904"}
--- PASS: TestRepeatedPOST (6.01s)
=== RUN   TestRunStop
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53970"}
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:133	httpjson input worker has started.	{"url": "http://127.0.0.1:53970"}
2020-06-16T20:53:23.963-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53970
2020-06-16T20:53:23.963-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:551	[ERR] GET http://127.0.0.1:53970 request failed: Get http://127.0.0.1:53970: context canceled
2020-06-16T20:53:23.963-0700	ERROR	[httpjson]	httpjson/input.go:138	failed to execute http client.Do: Get http://127.0.0.1:53970: context canceled	{"url": "http://127.0.0.1:53970"}
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:139	httpjson input worker has stopped.	{"url": "http://127.0.0.1:53970"}
--- PASS: TestRunStop (0.00s)
=== RUN   TestOAuth2
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53972"}
2020-06-16T20:53:23.964-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53971
2020-06-16T20:53:23.964-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53972
--- PASS: TestOAuth2 (0.00s)
PASS
ok  	github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson	9.214s
(base)

elasticmachine · 2020-06-03T21:52:44Z

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Expand to view the summary

Build stats

Build Cause: [Branch indexing]
Start Time: 2020-06-18T15:15:42.525+0000
Duration: 124 min 41 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8464
Skipped	1553
Total	10017

Steps errors

Expand to view the steps failures

Name: Make -C libbeat testsuite
- Description: make -C libbeat testsuite
- Duration: 6 min 7 sec
- Start Time: 2020-06-18T15:39:31.209+0000
- log

Log output

Expand to view the last 100 lines of log output

[2020-06-18T17:18:49.067Z] 	at org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution.cancel(TimeoutStepExecution.java:165)
[2020-06-18T17:18:49.067Z] 	at org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution.access$100(TimeoutStepExecution.java:38)
[2020-06-18T17:18:49.067Z] 	at org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution$1.run(TimeoutStepExecution.java:139)
[2020-06-18T17:18:49.067Z] 	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-06-18T17:18:49.067Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-06-18T17:18:49.067Z] 
[2020-06-18T17:18:49.083Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-06-18T17:18:49.267Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Lint
[2020-06-18T17:18:49.747Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Winlogbeat-oss
[2020-06-18T17:18:50.074Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-06-18T17:18:50.342Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-06-18T17:18:50.827Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Dockerlogbeat
[2020-06-18T17:18:51.350Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Journalbeat-oss
[2020-06-18T17:18:51.718Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-06-18T17:18:52.066Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-06-18T17:18:52.456Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Libbeat-oss
[2020-06-18T17:18:52.742Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-06-18T17:18:53.092Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-06-18T17:18:53.590Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-06-18T17:18:53.996Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Heartbeat-oss
[2020-06-18T17:18:54.467Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-06-18T17:18:54.841Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-06-18T17:18:55.109Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Libbeat-x-pack
[2020-06-18T17:18:55.339Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-06-18T17:18:55.661Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-06-18T17:18:55.908Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-06-18T17:18:56.187Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-06-18T17:18:56.551Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-Windows
[2020-06-18T17:18:56.775Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Packetbeat-oss
[2020-06-18T17:18:56.956Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-Windows
[2020-06-18T17:18:57.239Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-06-18T17:18:57.441Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-x-pack
[2020-06-18T17:18:57.659Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-06-18T17:18:57.848Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-oss
[2020-06-18T17:18:58.060Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-06-18T17:18:58.245Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-06-18T17:18:58.463Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-06-18T17:18:58.697Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-06-18T17:18:58.999Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-06-18T17:18:59.400Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-06-18T17:18:59.607Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-06-18T17:18:59.769Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-06-18T17:19:00.030Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-06-18T17:19:00.197Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-06-18T17:19:00.372Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-06-18T17:19:00.922Z] + cat
[2020-06-18T17:19:00.922Z] + /usr/local/bin/runbld ./runbld-script
[2020-06-18T17:19:00.922Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-06-18T17:19:07.692Z] runbld>>> runbld started
[2020-06-18T17:19:07.692Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-06-18T17:19:08.663Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18956' in order of occurrence in the config (last value wins).
[2020-06-18T17:19:10.102Z] runbld>>> Debug logging enabled.
[2020-06-18T17:19:10.102Z] runbld>>> Storing result
[2020-06-18T17:19:10.102Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-06-18T17:19:10.102Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200618171909-160AC007
[2020-06-18T17:19:10.102Z] runbld>>> Adding system facts.
[2020-06-18T17:19:10.692Z] runbld>>> Adding vcs info for the latest commit:  4657cbfc603647101b87550a45d0124e94ccbf92
[2020-06-18T17:19:10.692Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-06-18T17:19:10.971Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-06-18T17:19:10.971Z] + echo 'Processing JUnit reports with runbld...'
[2020-06-18T17:19:10.971Z] Processing JUnit reports with runbld...
[2020-06-18T17:19:11.262Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-06-18T17:19:11.262Z] runbld>>> DURATION: 29ms
[2020-06-18T17:19:11.262Z] runbld>>> STDOUT: 40 bytes
[2020-06-18T17:19:11.262Z] runbld>>> STDERR: 49 bytes
[2020-06-18T17:19:11.262Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-06-18T17:19:11.262Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats
[2020-06-18T17:19:12.691Z] runbld>>> Storing build metadata: 
[2020-06-18T17:19:12.691Z] runbld>>> Adding test report.
[2020-06-18T17:19:12.691Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats
[2020-06-18T17:19:14.103Z] runbld>>> Found 105 test output files
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-06-18T17:19:15.648Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-06-18T17:19:15.648Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-06-18T17:19:16.148Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 9867 Skipped: 1316
[2020-06-18T17:19:16.148Z] runbld>>> Storing result
[2020-06-18T17:19:16.148Z] runbld>>> FAILURES: 0
[2020-06-18T17:19:16.415Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-06-18T17:19:16.415Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200618171909-160AC007
[2020-06-18T17:19:16.415Z] runbld>>> Email notification disabled by environment variable.
[2020-06-18T17:19:16.415Z] runbld>>> Slack notification disabled by environment variable.
[2020-06-18T17:19:22.124Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956
[2020-06-18T17:19:22.285Z] [INFO] getVaultSecret: Getting secrets
[2020-06-18T17:19:22.358Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-18T17:19:23.479Z] + chmod 755 generate-build-data.sh
[2020-06-18T17:19:23.479Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/runs/18 ABORTED 7420646
[2020-06-18T17:19:23.479Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/runs/18/steps/?limit=10000 -o steps-info.json
[2020-06-18T17:19:25.962Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/runs/18/tests/?status=FAILED -o tests-errors.json
[2020-06-18T17:19:27.305Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/runs/18/log/ -o pipeline-log.txt

elasticmachine · 2020-06-03T21:55:46Z

Pinging @elastic/siem (Team:SIEM)

alakahakai · 2020-06-08T00:23:58Z

jenkins, test this

andrewkroh

Can you please update the documentation for the httpjson input to include these new options.

x-pack/filebeat/input/httpjson/config.go

x-pack/filebeat/input/httpjson/input.go

… retryablehttp-okta

alakahakai · 2020-06-18T22:40:15Z

@andrewkroh Please review again. thanks

andrewkroh

LGTM

andrewkroh · 2020-06-30T16:01:49Z

Please backport this to 7.8 and 7.x.

…put (elastic#18956) - Add automatic retries and exponential backoff to Filebeat httpjson input (elastic#18956) - Add test cases - Support the new OAuth2 functionality (cherry picked from commit 0dcb3df)

…put (#18956) (#19538) - Add automatic retries and exponential backoff to Filebeat httpjson input (#18956) - Add test cases - Support the new OAuth2 functionality (cherry picked from commit 0dcb3df) Co-authored-by: Lei Qiu <lei.qiu@elastic.co>

…put (elastic#18956) - Add automatic retries and exponential backoff to Filebeat httpjson input (elastic#18956) - Add test cases - Support the new OAuth2 functionality

alakahakai added Filebeat Filebeat Team:SIEM labels Jun 3, 2020

alakahakai added enhancement review labels Jun 3, 2020

alakahakai requested review from andrewkroh and leehinman June 3, 2020 21:57

andrewkroh requested changes Jun 8, 2020

View reviewed changes

x-pack/filebeat/input/httpjson/config.go Outdated Show resolved Hide resolved

x-pack/filebeat/input/httpjson/input.go Outdated Show resolved Hide resolved

x-pack/filebeat/input/httpjson/input.go Outdated Show resolved Hide resolved

andrewkroh requested a review from marc-gr June 8, 2020 23:03

Merge with master

bb39550

alakahakai requested a review from andrewkroh June 17, 2020 04:22

Lei Qiu added 8 commits June 17, 2020 09:44

Update build

81b277d

Fix merge problems

4fbfb16

Merge branch 'master' into retryablehttp-okta

6b55b3a

Run go mod to update go.sum

626a1e5

Merge branch 'retryablehttp-okta' of github.com:alakahakai/beats into…

c0f65e8

… retryablehttp-okta

Update test_xpack_modules.py

ff5ed88

Update test_xpack_modules.py

7ae5037

Update test_http_endpoint.py

0a1e04f

andrewkroh approved these changes Jun 18, 2020

View reviewed changes

alakahakai merged commit 0dcb3df into elastic:master Jun 19, 2020

andrewkroh added the needs_backport PR is waiting to be backported to other branches. label Jun 30, 2020

marc-gr mentioned this pull request Jul 1, 2020

Cherry-pick #18956 to 7.x: Add automatic retries and exponential backoff to Filebeat httpjson input #19538

Merged

marc-gr added v7.9.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jul 1, 2020

andrewkroh mentioned this pull request Aug 13, 2020

[Filebeat] Okta Module Crashing Frequently #18530

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add automatic retries and exponential backoff to Filebeat httpjson input #18956

Add automatic retries and exponential backoff to Filebeat httpjson input #18956

alakahakai commented Jun 3, 2020 •

edited

Loading

elasticmachine commented Jun 3, 2020 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jun 3, 2020

alakahakai commented Jun 8, 2020

andrewkroh left a comment

alakahakai commented Jun 18, 2020

andrewkroh left a comment

andrewkroh commented Jun 30, 2020 •

edited

Loading

Add automatic retries and exponential backoff to Filebeat httpjson input #18956

Add automatic retries and exponential backoff to Filebeat httpjson input #18956

Conversation

alakahakai commented Jun 3, 2020 • edited Loading

elasticmachine commented Jun 3, 2020 • edited Loading

❕ Build Aborted

Build stats

Test stats 🧪

Steps errors

Log output

elasticmachine commented Jun 3, 2020

alakahakai commented Jun 8, 2020

andrewkroh left a comment

Choose a reason for hiding this comment

alakahakai commented Jun 18, 2020

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh commented Jun 30, 2020 • edited Loading

alakahakai commented Jun 3, 2020 •

edited

Loading

elasticmachine commented Jun 3, 2020 •

edited

Loading

andrewkroh commented Jun 30, 2020 •

edited

Loading