Add awsfargate module to collect container logs from Amazon ECS on Fargate #25041

kaiyan-sheng · 2021-04-13T15:46:06Z

What does this PR do?

This PR is to add support for collecting logs from AWS Fargate with awsfargate module.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Step1: Follow steps in Metricbeat documentation to create some containers running using Fargate.

Step2: Check CloudWatch to get the log group ARN where your fargate logs are sent to.

Step3: Enable awsfargate module using ./filebeat modules enable awsfargate.

Step4: Modify awsfargate.yml with credentials and the log group ARN. For example:

- module: awsfargate
  log:
    enabled: true
    var.credential_profile_name: elastic-beats
    var.log_group_arn: arn:aws:logs:us-east-1:1234567890:log-group:/ecs/metricbeat-awsfargate:*

Step5: Start Filebeat and you should be able to see logs getting ingested into ES.

Related issues

Closes [Logs] Add Fargate support integrations#559

elasticmachine · 2021-04-13T15:46:29Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-04-13T15:46:29Z

Pinging @elastic/integrations (Team:Platforms)

botelastic · 2021-04-13T15:46:43Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2021-04-13T16:43:14Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25041 updated
Start Time: 2021-04-20T04:04:55.351+0000
Duration: 56 min 27 sec
Commit: bf0a8b2

Test stats 🧪

Test	Results
Failed	0
Passed	13609
Skipped	2271
Total	15880

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	13609
Skipped	2271
Total	15880

ChrsMark

Looks great!

One question: Is there any way we can enrich this logs with container metadata?

filebeat/docs/modules/awsfargate.asciidoc

kaiyan-sheng · 2021-04-15T15:16:04Z

One question: Is there any way we can enrich this logs with container metadata?

@ChrsMark Thanks for the review!! I'm relying on the add_docker_metadata processor to add container info container.id and this can be used to connect logs from Filebeat with monitoring metrics from awsfargate Metricbeat module.

ChrsMark · 2021-04-19T07:25:22Z

One question: Is there any way we can enrich this logs with container metadata?

@ChrsMark Thanks for the review!! I'm relying on the add_docker_metadata processor to add container info container.id and this can be used to connect logs from Filebeat with monitoring metrics from awsfargate Metricbeat module.

Hmm, add_docker_metadata requires access to docker socket, not sure if it's gonna work here since the setup is different. Or do I miss something 🤔 ?

ChrsMark

lgtm

ChrsMark · 2021-04-19T07:25:44Z

/test

kaiyan-sheng · 2021-04-19T13:37:15Z

CI failure should be fixed by #25141

exekias · 2021-04-19T13:57:42Z

x-pack/filebeat/module/awsfargate/log/ingest/pipeline.yml

+      field: message
+      target_field: event.original
+      ignore_missing: true


Could you provide some detail on why doing this rename?

Ahh sorry!! I thought we are moving all message to event.oiriginal. But I see #14708 is the other way around. I will remove this renaming here.

exekias · 2021-04-19T13:58:31Z

x-pack/filebeat/module/awsfargate/log/test/nginx.log-expected.json

+    {
+        "event.dataset": "awsfargate.log",
+        "event.module": "awsfargate",
+        "event.original": "10.0.1.195 - - [24/Sep/2020:16:45:19 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"ELB-HealthChecker/2.0\" \"-\"",
+        "fileset.name": "log",
+        "input.type": "log",
+        "log.offset": 189,
+        "service.type": "awsfargate"
+    }


I'm wondering, is cloudwatch providing any extra meta about the source of the logs? I would expect some info about the containers generating these logs

Hmm the only metadata I can find is the container ID, which is in the name of the log stream. For example: ecs/metricbeat-awsfargate/397eb2787a7d4f7783d03c49cafd244c.

mergify · 2021-04-19T22:21:31Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fargate_logs upstream/fargate_logs
git merge upstream/master
git push upstream fargate_logs

… logs from Amazon ECS on Fargate (#25161) * Add awsfargate module to collect container logs from Amazon ECS on Fargate (#25041) (cherry picked from commit 9013d07)

…-github-pr-comment-template * upstream/master: [Ingest Manager] Keep http and logging config during enroll (elastic#25132) Refactor kubernetes autodiscover to avoid skipping short-living pods (elastic#24742) [libbeat] New decode xml wineventlog processor (elastic#25115) Add svc to agent k8s clusterRole (elastic#25146) Add awsfargate module to collect container logs from Amazon ECS on Fargate (elastic#25041) [Filebeat][Cisco ASA] log enhancement and performance (elastic#24744) Watch kubernetes namespaces for autodiscover metadata for pods (elastic#25117) Cyberark Privileged Access Security module (elastic#24803) [Elastic Agent] Log the container command output with LOGS_PATH (elastic#25150) Fix for tests after `device...` field has been removed (elastic#25141) [Ingest Manager] Restart process on output change (elastic#24907) Set --insecure in container when FLEET_SERVER_ENABLE and FLEET_INSECURE set. (elastic#25137) [filebeat] Update documentation / changelog / beta warnings for the syslog input (elastic#25047) Add support for ignore_inactive in filestream input (elastic#25036) Fix bug with annotations dedot config on k8s not used (elastic#25111)

kaiyan-sheng added 3 commits April 12, 2021 23:35

add fargate fileset in aws module to collect ECS container logs

9912d18

add test file

0e1f53e

create awsfargate module in Filebeat and log fileset

171f333

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 13, 2021

kaiyan-sheng self-assigned this Apr 13, 2021

kaiyan-sheng added Team:Integrations Label for the Integrations team Team:Platforms Label for the Integrations - Platforms team labels Apr 13, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 13, 2021

kaiyan-sheng added needs_team Indicates that the issue/PR needs a Team:* label test-plan Add this PR to be manual test plan and removed Team:Platforms Label for the Integrations - Platforms team labels Apr 13, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 13, 2021

kaiyan-sheng added the in progress Pull request is currently in progress. label Apr 13, 2021

fix awsfargate config

8c07883

ChrsMark reviewed Apr 15, 2021

View reviewed changes

filebeat/docs/modules/awsfargate.asciidoc Outdated Show resolved Hide resolved

add fields.yml for log fileset

f123a3e

kaiyan-sheng requested a review from ChrsMark April 15, 2021 15:24

kaiyan-sheng added 3 commits April 15, 2021 23:39

remove duplicate definition of aws-credentials-options

db8bc66

change ref id to awsfargate-credentials

bd0566b

run mage update

db42e3c

ChrsMark approved these changes Apr 19, 2021

View reviewed changes

Merge remote-tracking branch 'upstream/master' into fargate_logs

f02fd9f

exekias reviewed Apr 19, 2021

View reviewed changes

kaiyan-sheng added 4 commits April 20, 2021 06:26

Merge remote-tracking branch 'upstream/master' into fargate_logs

3eb6391

remove rename processor

7ac33e5

rerun mage test

3327930

fix zeek generated file

bf0a8b2

kaiyan-sheng merged commit 9013d07 into elastic:master Apr 20, 2021

kaiyan-sheng deleted the fargate_logs branch April 20, 2021 07:17

kaiyan-sheng mentioned this pull request Apr 20, 2021

Cherry-pick #25041 to 7.x: Add awsfargate module to collect container logs from Amazon ECS on Fargate #25161

Merged

6 tasks

kaiyan-sheng added the v7.13.0 label Apr 20, 2021

andresrc added the test-plan-added This PR has been added to the test plan label Apr 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add awsfargate module to collect container logs from Amazon ECS on Fargate #25041

Add awsfargate module to collect container logs from Amazon ECS on Fargate #25041

kaiyan-sheng commented Apr 13, 2021 •

edited

Loading

elasticmachine commented Apr 13, 2021

elasticmachine commented Apr 13, 2021

botelastic bot commented Apr 13, 2021

elasticmachine commented Apr 13, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

ChrsMark left a comment

kaiyan-sheng commented Apr 15, 2021

ChrsMark commented Apr 19, 2021 •

edited

Loading

ChrsMark left a comment

ChrsMark commented Apr 19, 2021

kaiyan-sheng commented Apr 19, 2021

exekias Apr 19, 2021

kaiyan-sheng Apr 19, 2021

exekias Apr 19, 2021

kaiyan-sheng Apr 19, 2021

mergify bot commented Apr 19, 2021

Add awsfargate module to collect container logs from Amazon ECS on Fargate #25041

Add awsfargate module to collect container logs from Amazon ECS on Fargate #25041

Conversation

kaiyan-sheng commented Apr 13, 2021 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Apr 13, 2021

elasticmachine commented Apr 13, 2021

botelastic bot commented Apr 13, 2021

elasticmachine commented Apr 13, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

ChrsMark left a comment

Choose a reason for hiding this comment

kaiyan-sheng commented Apr 15, 2021

ChrsMark commented Apr 19, 2021 • edited Loading

ChrsMark left a comment

Choose a reason for hiding this comment

ChrsMark commented Apr 19, 2021

kaiyan-sheng commented Apr 19, 2021

exekias Apr 19, 2021

Choose a reason for hiding this comment

kaiyan-sheng Apr 19, 2021

Choose a reason for hiding this comment

exekias Apr 19, 2021

Choose a reason for hiding this comment

kaiyan-sheng Apr 19, 2021

Choose a reason for hiding this comment

mergify bot commented Apr 19, 2021

kaiyan-sheng commented Apr 13, 2021 •

edited

Loading

elasticmachine commented Apr 13, 2021 •

edited by jenkins-beats-ci bot

Loading

ChrsMark commented Apr 19, 2021 •

edited

Loading