[Filebeat] Fix s3 input parsing json file without expand_event_list_from_field

[kaiyan-sheng]

What does this PR do?

This PR is to fix s3 input when parsing json files without expand_event_list_from_field config parameter.
During testing I found offset is not working properly for s3 input events, this PR also fix it.

Why is it important?

For some logs, such as Cloudflare, json looks like:

{"id": "0001", "hey": "there", "how": {"are": "you"}}
{"id": "0002", "hope": "you", "are": {"doing": "well"}}
{"id": "0003", "I": "am", "doing": {"O": "K"}}

instead of with a head field like:

{
  "Records": [
    {
      "eventVersion": "1.05",
      "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROAJ3UFBPIHDEKKCOI2G:AssumeRoleSession",
        "arn": "arn:aws:sts::123456789012:assumed-role/CloudHealth/AssumeRoleSession",
        "accountId": "123456789012"
      }
]

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Create a test log file:

$ cat s3filebeat.log 
{"id": "0001", "hey": "there", "how": {"are": "you"}}
{"id": "0002", "hope": "you", "are": {"doing": "well"}}
{"id": "0003", "I": "am", "doing": {"O": "K"}}

Gzip the log file:

gzip s3filebeat.log 

Upload the file to S3 bucket and add property:

$ aws --profile PROFILE s3api put-object --body ./s3filebeat.log.gz --bucket lucas-test-filebeat-s3 --content-encoding gzip --content-type application/json --key s3filebeat.log.gz
{
    "ETag": "\"955ed9f01b6ee38dbba167daab9ebbbb\""
}

or manually upload this file to s3 bucket and change the property there.

Change filebeat input to s3 in filebeat.yml

filebeat.inputs:
- type: s3
  queue_url: https://sqs.us-east-1.amazonaws.com/428152502467/test-fb-ks
  credential_profile_name: elastic-beats

Run ./filebeat -e

Related issues

• Closes Filebeat S3 input plugin cannot parse jsonl file with content-type set as application/json #19902

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[lag13]

I just tested this and it works for my use case! Thanks again for working on this @kaiyan-sheng

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #19962 updated]

• Start Time: 2020-07-21T17:46:10.899+0000

• Duration: 51 min 24 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2434
Skipped	385
Total	2819