Collector config not resolving resource attributes when used in log_stream_name

[davekant]

We are using our own custom config file for the aws-otel-collector and trying to configure the name of the log-stream using the following approach.
otel-container-metrics-config.yaml.txt

exporters:
awsemf:
namespace: ECS/ContainerInsights/Otel
log_group_name: '/aws/ecs/containerinsights/{ClusterName}/performance'
log_stream_name: '{TaskDefinitionFamily}-{TaskId}'

However, only the TaskId is being resolved when viewing the log stream in cloudwatch logs (attached).

We also tried using ServiceName and ContainerName but had similar behaviour.
In CloudWatch Metrics, these values are being resolved in both the custom and default dimensions.

[hossain-rayhan]

This is expected behavior. For the workloads running on ECS, only the {ClusterName} and {TaskId} can be used as valid placeholder for log_group and log_stream name. Recently we added support for {ContainerInstanceId} in the log_stream name. Please check the README for details.

[davekant]

From a usability perspective, if there are many services running in the cluster it can be difficult to identify all the log streams associated with a single service. Many of our services are auto-scaled and so the number of tasks running can exceed 100+. The CloudWatch Log Streams filter is difficult to use to filter the streams because its name based and taskIds are not human friendly. However, a human readable log stream name (such as {TaskDefinitionFamily}-{TaskId} ) would be easier to work with.

Currently, we are using log insights to analyse the @message for the TaskDefinitionFamily.

[hossain-rayhan]

Hi @davekant , thanks for opening this. I understand this would be useful. We do have the TaskDefinitionFamily name and it's possible to support it. I added an item in our backlog. We will start working on this soon when I get some cycle.

Another thing, how important the TaskDefinitionRevision is in your opinion?

[aateeqi]

I'd like to pick this up

[hossain-rayhan]

@aateeqi thanks. You can send a PR. I will help with review.