Cherry-pick #22036 to 7.10: [Filebeat] Check context.Canceled and fix s3 input config

[kaiyan-sheng]

Cherry-pick of PR #22036 to 7.10 branch. Original message:

What does this PR do?

This PR fixes s3 input config in filebeat.reference.yml and also remove error message when Filebeat is stopped.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Logs

kaiyansheng@KaiyanMacBookPro:~/go/src/github.com/elastic/beats/x-pack/filebeat (s3_bug)$ ./filebeat -e
2020-10-20T13:27:29.449-0600    INFO    instance/beat.go:633    Home path: [/Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat] Config path: [/Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat] Data path: [/Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat/data] Logs path: [/Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat/logs]
2020-10-20T13:27:29.449-0600    INFO    instance/beat.go:641    Beat ID: e60e9f14-e718-4bef-80e6-d573fdd4c207
2020-10-20T13:27:29.467-0600    INFO    [beat]  instance/beat.go:965    Beat info       {"system_info": {"beat": {"path": {"config": "/Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat", "data": "/Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat/data", "home": "/Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat", "logs": "/Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat/logs"}, "type": "filebeat", "uuid": "e60e9f14-e718-4bef-80e6-d573fdd4c207"}}}
2020-10-20T13:27:29.467-0600    INFO    [beat]  instance/beat.go:974    Build info      {"system_info": {"build": {"commit": "a10dca7959a5c09391e853d6e8d3e45bbee0b10f", "libbeat": "8.0.0", "time": "2020-10-20T19:27:02.000Z", "version": "8.0.0"}}}
2020-10-20T13:27:29.467-0600    INFO    [beat]  instance/beat.go:977    Go runtime info {"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":12,"version":"go1.14.4"}}}
2020-10-20T13:27:29.468-0600    INFO    [beat]  instance/beat.go:981    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-10-01T13:09:29.78477-06:00","name":"KaiyanMacBookPro","ip":["127.0.0.1/8","::1/128","fe80::1/64","fe80::aede:48ff:fe00:1122/64","fe80::405:380d:35df:35e8/64","192.168.0.5/24","fe80::14bb:6cff:fe27:58d4/64","fe80::14bb:6cff:fe27:58d4/64","fe80::43d9:979e:a3a0:19d9/64","fe80::a7e7:f04d:9e00:ceba/64"],"kernel_version":"19.6.0","mac":["ac:de:48:00:11:22","f2:18:98:75:78:55","f0:18:98:75:78:55","82:21:7f:03:d8:01","82:21:7f:03:d8:00","82:21:7f:03:d8:05","82:21:7f:03:d8:04","82:21:7f:03:d8:01","02:18:98:75:78:55","16:bb:6c:27:58:d4","16:bb:6c:27:58:d4"],"os":{"family":"darwin","platform":"darwin","name":"Mac OS X","version":"10.15.6","major":10,"minor":15,"patch":6,"build":"19G2021"},"timezone":"MDT","timezone_offset_sec":-21600,"id":"9C7FAB7B-29D1-5926-8E84-158A9CA3E25D"}}}
2020-10-20T13:27:29.468-0600    INFO    [beat]  instance/beat.go:1010   Process info    {"system_info": {"process": {"cwd": "/Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat", "exe": "./filebeat", "name": "filebeat", "pid": 71067, "ppid": 14270, "start_time": "2020-10-20T13:27:28.640-0600"}}}
2020-10-20T13:27:29.468-0600    INFO    instance/beat.go:298    Setup Beat: filebeat; Version: 8.0.0
2020-10-20T13:27:29.468-0600    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-8.0.0' as ILM is enabled.
2020-10-20T13:27:29.468-0600    INFO    eslegclient/connection.go:99    elasticsearch url: http://localhost:9200
2020-10-20T13:27:29.469-0600    INFO    [publisher]     pipeline/module.go:113  Beat name: KaiyanMacBookPro
2020-10-20T13:27:29.469-0600    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-10-20T13:27:29.469-0600    INFO    instance/beat.go:454    filebeat start running.
2020-10-20T13:27:29.470-0600    INFO    memlog/store.go:119     Loading data file of '/Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2020-10-20T13:27:29.470-0600    INFO    memlog/store.go:124     Finished loading transaction log file for '/Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat/data/registry/filebeat'. Active transaction id=0
2020-10-20T13:27:29.470-0600    INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 0
2020-10-20T13:27:29.470-0600    INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 2
2020-10-20T13:27:29.470-0600    WARN    [input] v2/loader.go:104        BETA: The s3 input is beta      {"input": "s3", "stability": "Beta", "deprecated": false}
2020-10-20T13:27:29.471-0600    INFO    [crawler]       beater/crawler.go:141   Starting input (ID: 13817434867768840760)
2020-10-20T13:27:29.471-0600    INFO    [input.s3]      compat/compat.go:110    Input s3 starting
2020-10-20T13:27:29.471-0600    INFO    [crawler]       beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 1
2020-10-20T13:27:29.471-0600    INFO    cfgfile/reload.go:164   Config reloader started
2020-10-20T13:27:29.471-0600    INFO    cfgfile/reload.go:224   Loading of config files completed.
2020-10-20T13:27:29.472-0600    INFO    [input.s3]      s3/input.go:107 visibility timeout is set to 300 seconds        {"queue_url": "https://sqs.us-east-1.amazonaws.com/428152502467/test-fb-ks", "region": "us-east-1"}
2020-10-20T13:27:29.472-0600    INFO    [input.s3]      s3/input.go:108 aws api timeout is set to 2m0s  {"queue_url": "https://sqs.us-east-1.amazonaws.com/428152502467/test-fb-ks", "region": "us-east-1"}
2020-10-20T13:27:29.472-0600    INFO    [input.s3]      s3/collector.go:93      s3 input worker has started.    {"queue_url": "https://sqs.us-east-1.amazonaws.com/428152502467/test-fb-ks", "region": "us-east-1"}
2020-10-20T13:27:32.450-0600    INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:101    add_cloud_metadata: hosting provider type not detected.

^C2020-10-20T13:27:54.319-0600  INFO    beater/filebeat.go:422  Stopping filebeat
2020-10-20T13:27:54.319-0600    INFO    beater/crawler.go:148   Stopping Crawler
2020-10-20T13:27:54.319-0600    INFO    beater/crawler.go:158   Stopping 1 inputs
2020-10-20T13:27:54.319-0600    INFO    cfgfile/reload.go:227   Dynamic config reloader stopped
2020-10-20T13:27:54.319-0600    INFO    [crawler]       beater/crawler.go:163   Stopping input: 13817434867768840760
2020-10-20T13:27:54.320-0600    INFO    [input.s3]      compat/compat.go:131    Input 's3' stopped
2020-10-20T13:27:54.320-0600    INFO    beater/crawler.go:178   Crawler stopped
2020-10-20T13:27:54.320-0600    INFO    [registrar]     registrar/registrar.go:132      Stopping Registrar
2020-10-20T13:27:54.320-0600    INFO    [registrar]     registrar/registrar.go:166      Ending Registrar
2020-10-20T13:27:54.320-0600    INFO    [registrar]     registrar/registrar.go:137      Registrar stopped
2020-10-20T13:27:54.320-0600    INFO    [input.s3]      s3/collector.go:113     s3 input worker has stopped.    {"queue_url": "https://sqs.us-east-1.amazonaws.com/428152502467/test-fb-ks", "region": "us-east-1"}
2020-10-20T13:27:54.320-0600    INFO    [input.s3]      compat/compat.go:123    Input 's3' stopped
2020-10-20T13:27:54.328-0600    INFO    [monitoring]    log/log.go:153  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":43,"time":{"ms":43}},"total":{"ticks":189,"time":{"ms":189},"value":189},"user":{"ticks":146,"time":{"ms":146}}},"info":{"ephemeral_id":"c6d1a7ab-ebc3-4dd0-b772-9871f32a0676","uptime":{"ms":24921}},"memstats":{"gc_next":18225712,"memory_alloc":13896112,"memory_total":49105456,"rss":57323520},"runtime":{"goroutines":17}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":12},"load":{"1":2.8237,"15":2.7939,"5":2.876,"norm":{"1":0.2353,"15":0.2328,"5":0.2397}}}}}}
2020-10-20T13:27:54.328-0600    INFO    [monitoring]    log/log.go:154  Uptime: 24.922236644s
2020-10-20T13:27:54.328-0600    INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2020-10-20T13:27:54.328-0600    INFO    instance/beat.go:460    filebeat stopped.

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: [Pull request #22115 opened]

• Start Time: 2020-10-22T21:53:17.916+0000

• Duration: 68 min 22 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1947
Skipped	259
Total	2206