Feature: Create CloudWatch Log groups for Functions

[theburningmonk]

Description:

CloudWatch Log Groups are not deleted when a stack is deleted. Leaving behind lots of abandoned log groups in CloudWatch Logs.

This happens because SAM doesn't explicitly create the log groups in the generated CloudFormation stack, and rely on Lambda to generate them at runtime, so the log groups are not part of the stack and don't get deleted when the stack is deleted.

Steps to reproduce the issue:

1. Create a template with Lambda functions
2. Package
3. Deploy
4. Invoke the functions a few times
5. Delete stack

Observed result:
The log groups remain in the account.

Expected result:
The log groups should be deleted.

[jlhood]

@theburningmonk Thanks for reporting this issue! We're looking at adding SAM support to give users more control over the log group associated with their Lambda function. We could add this as part of that feature although I think it would need to be an optional setting since some customers may want to leave log groups around even in the event of a function deletion for compliance reasons.

[53ningen]

(Just FYI)
I often define CloudWatch Logs resources explicitly along with Lambda function resources like this:

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
  HelloWorldFunctionLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
        LogGroupName: !Sub /aws/lambda/${HelloWorldFunction}
        RetentionInDays: 7

If you delete a stack which was created by using the template above, a log group will be also deleted.
I think this could be easy workaround for now.

[jlhood]

@53ningen Thanks for the workaround. There are some caveats to using it that I want to call out:

The way you have it, you're allowing CloudFormation to name your Lambda function and then referencing it when creating the CW Logs group. This opens a race condition where if the function is invoked prior to the CW Logs group being created by CloudFormation, the CloudFormation deployment will fail with a log group already exists error. We've seen this happen, for example, when adding a new Lambda function that listens on an existing DynamoDB/Kinesis stream, SNS topic, or SQS queue.

A way to close this race condition is to explicitly name the function and then add a DependsOn to the LogGroup to ensure it gets created first. This is not ideal since you now have to name your own function (which has its own issues), but it does ensure this race condition does not happen.

Again, not against people using the workaround, just wanted to call out pitfalls.

Internally, we're discussing a way for SAM to support this natively without these pitfalls. My ideal is to add logging properties to AWS::Serverless::Function that make it possible to configure properties on the underlying LogGroup and have SAM handle all of these race conditions auto-magically for you under the covers. Hoping to post an RFC issue for that in the next couple months.

[brysontyrrell]

@jlhood I'd like to +1 on configuring logging as a property of the AWS::Serverless::Function.

MyFunction:
  Type: AWS::Serverless::Function
  Properties:
    Logging:
      LogGroup: !Ref MyLogGroup  # Optional: use an existing CloudWatch Log Group and do not automatically create one (cannot be combined with other properties)
      LogGroupName: /my/log/group/name  # Optional:provide your own name for the log group
      RetentionInDays: 7  # Optional: define the number of days to retain logs (default to 30 when not defined as with other AWS services)
      Retain: true  # Optional: preserve log group on resource teardown (default to `false`)

[praneetap]

@brysontyrrell This would be really cool to add, thanks for the spec proposal.

[rusharma]

Also observing this issue on a stack created using CDK.
Log groups are left behind when we redeploy the stacks using "cdk deploy". Also, not just for lambda functions, this is also true for fargate cluster logs.

[jungseoklee]

@53ningen @jlhood +1 to add logging as property of AWS::Serverless::Function. I would like to configure retention period of CloudWatch Logs because "Never Expire" option is not cost-efficient.

[red8888]

is there no workaround? I have 100s of log groups I have to clean up. is it possible to manually provision a log group in the same template and reference it in the lambda resource?

[keetonian]

@red8888 yes. There are a few approaches for this.

Before 1st invocation of a Lambda function:

Before a Lambda function is invoked for the first time, it is possible to include a AWS::Logs::LogGroup resource in your template that is configured in the way Lambda expects. If this is attempted after a Lambda function is invoked, it will fail with an error from CloudFormation, and the next approach should be used.

  LambdaLogGroup:
    DependsOn: MyLambdaFunction # the Lambda function logical id
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/lambda/${MyLambdaFunction}" # Default lambda log group naming format

After 1st invocation of a Lambda function:

If a Lambda function has been invoked, then it will have automatically created a LogGroup in CloudWatch. At this point, you would need to use CloudFormation resource import to import it into the template. Make sure to configure the log group in the template how you want it to be configured in CloudWatch (including following the same naming convention), since it will take over updating and configuring this resource in the future.

Deleting an existing log group:

I don't have a good solution for this in a production service or account. I usually run a script to clean up other environments, but more care should be taken when deleting logs from prod.

[red8888]

The following does not work:

  LambdaLogGroup:
    DependsOn: MyLambdaFunction # the Lambda function logical id
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/lambda/${MyLambdaFunction}" 

I deploy the sam template/stack. See the log groups are created and the functions are using them- no function has been invoked yet. I delete the stack and they are still there.

This really sucks, I do lots of branch deploys of isolated template deployments, each with multiple functions so I have crap tons of log groups I have to clean up constantly

If I remove DependsOn I can duplicate log groups (the ones created by sam and the ones defined by me in the template) and when I delete the stack the ones automatically created are left behind. If I have DependsOn I dont have the duplicate log groups problem but then when I delete the stack they arent removed

[yiyange]

I see this post https://operatingops.com/2019/09/28/cloudwatch-logs-preventing-orphaned-log-groups/ talking about orphaned log groups. it seems that it has to do with the policies one associates with the lambda. Specifically it is saying you should not grant lambda this action logs:CreateLogGroup. Maybe worth trying (if that is the case)?

[tuukkamustonen]

Removing logs:CreateLogGroup permissions (as suggested by @yiyange above) fixed it for me.

Now, when the Log Group has just been deleted and if Lambda function runs one last time or two, logs just get to dev null. That should be fine, as we are deleting the function in the first place.

[jplock]

I’ve been trying to create log groups in my CF templates and following the policy information in https://typicalrunt.me/2019/09/20/enforcing-least-privilege-when-logging-lambda-functions-to-cloudwatch/

but yea, don’t grant the lambda execution role “logs:CreateLogGroup”

[sanosuke-s74]

I would love to see this expanded to provide the option to use an existing log group for any lambda. Currently, I have a step function with 6 lambdas, and it would be cool to have all lambdas point to the same log group for easy searching across them all

[ryanblock]

Just adding my voice to the folks here calling for the ability to add log customization SAM-provisioned Lambdas. I recognize this is a challenge, but excellent logging hygiene is extremely important (especially since customer data sometimes has a way of finding its way into logs!).

We would very much like to add log retention functionality to @architect provisioned resources. I'm not entirely surprised this has been discussed since fairly early on (2018 and before, see #257), and hope y'all can find a way to make it happen!

[gwlester]

This issue is over three years old -- is there any solution on the horizon???

[sayandcode]

Just realized this issue still exists, when checking my Cloudwatch logs

[GuidoNebiolo]

Reported also in Ideas: #2665

[jfuss]

Duplicate of #2665. Linking them together but please more further conversations to the discussion.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.