ILambdaLogger should support parameterized and structured logging

[dhhoang]

Describe the feature

The dotnet6 Lambda runtime should support parameterized and structured logging methods.

Note: This issue continues the conversation in #1232. We believe structured logging would benefit customers, but we understand that we want to minimize the dependencies of Amazon.Lambda.Core. So we are proposing a different approach.
Related issues:

• Allow log message formatting to be controlled programmatically #1000
• Amazon.Lambda.Logging.AspNetCore should support structural json logging #1144
• Microsoft.Extensions.Logging.ILogger interface support for Lambda #1232

Use Case

Currently, the ILambdaLogger interface only supports writing a message string to the log. While this is convienent for simple logging, sometimes it is desired to write logs in a structured format (a.k.a structured logging) to support processing and querying. For example, writing logs in JSON will allow users to have flexible queries with CloudWatch Logs insight. The current interface makes it difficult to do so because there is no support for parameterized logging.

An example of parameterized logging is the methods defined for Microsoft's ILogger: https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.Extensions.Logging.Abstractions/src/LoggerExtensions.cs.

Another benefit of parameterized logging is efficiency. Currently, ILambdaLogger requires forming log message strings, regardless of the configured log level. For example, even when we configure AWS_LAMBDA_HANDLER_LOG_LEVEL to Warning, Debug and Info messages still need to be formed. By supporting parameterized logging methods, log forming actions can be skipped when the corresponding log level is disabled.

Proposed Solution

First, an additional LogEntry method is added to the ILambdaLogger interface.

public interface ILambdaLogger
{
  /// <summary>
  /// Logs an entry.
  /// </summary>
  /// <typeparam name="TState">The type of the log entry.</typeparam>
  /// <param name="level">Log level.</param>
  /// <param name="entry">The log entry.</param>
  void LogEntry<TEntry>(LogLevel level, TEntry entry);
}

The purpose of this method is to have a generic way of writing a log entry with state data attached to it. The level specifies the log level. The entry parameter is what contains the log entry's properties, which are not only the log message, but also other state information, such as key-value pair data, occured exception etc. TEntry is made generic on purpose to allow different handlings based on different types of logs. For example, the MessageEntry defined below would be one type of log, but if in the future we want to support CloudWatch EMF format, a different log type can be defined and handled separately.

To support structured logging, a MessageEntry structure, which represents a typical log message, is defined as follows.

internal readonly struct MessageEntry
{
  /// <summary>
  /// The state data of the log entry.
  /// </summary>
  public IReadOnlyList<KeyValuePair<string, object>> State { get; }

  /// <summary>
  /// The exception included in the entry. This property is NULL if the log does not contain exception.
  /// </summary>
  public Exception Exception { get; }
  
  /// <summary>
  /// Gets the log message.
  /// </summary>
  public override string ToString();
}

When the implementation of ILambdaLogger sees a MessageEntry, it serializes and writes the entry to the output. The specific serialization will be defined by AWS_LAMBDA_HANDLER_LOG_FORMAT. For currently-defined options (Default and Unformatted) the output format will remain the same. We propose another option named JSON, which writes output in the following (JSON) format:

{
  "Level": "Information",
  "Message": "Hello Bar",   // returned by MessageEntry.ToString()
  "Format": "Hello {Foo}",  // the message format
  "Exception": null,        // the (formatted) exception and stack-trace
  "State": {                // the key-value pairs stored in MessageEntry.State
    "Foo": "Bar"           
  }
}

Parameterized logging

To make it easier to write structured logs, Amazon.Lambda.Core also provides another set of extension methods to support parameterized logging.

public static class LambdaLoggerExtensions 
{
  void Trace(this ILambdaLogger logger, string format, params object?[] args);
  void Debug(this ILambdaLogger logger, string format, params object?[] args);
  void Info(this ILambdaLogger logger, string format, params object?[] args);
  void Warning(this ILambdaLogger logger, string format, params object?[] args);
  void Error(this ILambdaLogger logger, string format, params object?[] args);
  void Error(this ILambdaLogger logger, Exception exception, string format, params object?[] args);
  void Critical(this ILambdaLogger logger, string format, params object?[] args);
  void Critical(this ILambdaLogger logger, Exception exception, string format, params object?[] args);
}

These methods will perform the follow actions:

• Form the MessageEntry based on parameters. The MessageEntry should only be a holder object which is cheap to create, and should not perform the message formatting itself.
• Call ILambdaLogger.LogEntry() with the appropriate log level and the formed entry. When a formatted log message is required, ILambdaLogger invokes MessageEntry.ToString(), at which point the message is formed. ILambdaLogger also has access to other state information to support structured logging.

For example, the following code the JSON sample output above:

string paramValue = "Bar";
logger.Info("Hello {Foo}", paramValue);

Other Information

Why introduce LambdaLoggerExtensions instead of including all parameterized methods in ILambdaLogger?

The ILambdaLogger.LogEntry is designed to be generic, to allow extensions later if required. Introducing other parameterized methods to ILambdaLogger would pollute the interface and imply that those methods do different things, while in fact, they all delegate to ILambdaLogger.LogEntry.

Idea: support CloudWatch EMF log format?

Emitting metrics using CloudWatch EMF log format from Lambda is a common and supported use case. With the APIs defined in this proposal we can support EMF logs by defining a new entry type EmfEntry, and handle it in the ILambdaLogger.LogEntry:

void LogEntry<TEntry>(LogLevel level, TEntry entry)
{
  switch(entry)
  {
    case MessageEntry messageEntry:
      WriteMessage(messageEntry);
      break;
    case EmfEntry emfEntry:
      WriteEmf(messageEntry);
      break;
  }
}

Specific details for this will probably be in a future proposal (if this one got accepted) but I hope this clarifies the purpose of defining a generic log interface :).

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

AWS .NET SDK and/or Package version used

"Amazon.Lambda.Core" Version="2.1.0"

Targeted .NET Platform

.NET 6

Operating System and version

Amazon Linux

[dhhoang]

@ashishdhingra , is there a process for reviewing the proposal? I'd be happy to send a PR with sample implementation if needed :)

[ashishdhingra]

@ashishdhingra , is there a process for reviewing the proposal? I'd be happy to send a PR with sample implementation if needed :)

@dhhoang Sure. PR would be helpful to be reviewed by the team.

[normj]

Closing as this work is being taken care of a larger initiative. Thanks @dhhoang and your work was included in the larger effort. Currently looking for feedback on the larger effort. #1747

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.