Adding custom attributes to the logs via the agent

[rajpawar02]

I am addressing the collection of application logs by New Relic APM agents.

I have referred this document to configure APM agents on one of the application.

However, logs add five attributes by default, namely entity.guid, entity.name, hostname, trace.id, and span.id.

Because we are running several services/containers for the same application, it is extremely difficult to identify a specific ECS service or container log using the above attributes.

Can we use the Ruby APM agent to add custom attributes like container_id, ecs_service_name, and so on? If we do so, it will be extremely beneficial.
Please let us know what you think about this.
Thanks

[kaylareopelle]

Hi @rajpawar02! 👋

Thanks for submitting this feature request. We're intrigued by your idea to add custom attributes to log events and would like to learn more about how you would like this feature to behave.

I'm hoping you can provide some insight for us on a few questions:

• How do you expect this feature to behave?

  • For example, we have the ability to add custom attributes to transaction traces by calling the method NewRelic::Agent#add_custom_attributes and to span events using the method NewRelic::Agent#add_custom_span_attributes. Would a method similar to these achieve your goal?

  • Alternatively, we have the ability to set the app name (entity.name in the default attributes) and hostname by configuration in either the newrelic.yml file or by environment variable. Would you prefer to set attributes like container_id and ecs_service_name using this strategy?

• Is there a specific list of attributes you would always like to have added to a log event?

  • If this is the case, what are your thoughts about having your desired attributes included in log events by default instead of as custom attributes? This feature wouldn't necessarily have to replace custom attributes. Rather, it's helpful for us to know if there's linking metadata customers expect to have on events that we do not currently support

• Are the custom attributes you're hoping to add static values that could be determined at startup, dynamic attributes that should be added at runtime, or both?

Your responses to these questions would be very helpful! We're investigating this prospect and may have more questions for you as our efforts continue.

Thanks for your support!

[rajpawar02]

Please see my answer to your question below.

How do you expect this feature to behave?

I appreciate both approaches, but I'd prefer to be able to specify generalized custom attributes like ecs service name and container id using an environment variable or the newrelic.yml file.

Is there a specific list of attributes you would always like to have added to a log event?

Because we operate the majority of our applications on the AWS ECS service, having the following attribute with log will be beneficial.

• ecs_cluster_name
• ecs_service_name
• ecs_task_name
• container_name
• task_id
• container_id

Are the custom attributes you're hoping to add static values that could be determined at startup, dynamic attributes that should be added at runtime, or both?

Some properties, such as ecs service name and ecs cluster name, are static. Some are dynamic, such as task id and container id, which are generated once the container is operating.

[kaylareopelle]

Thank you for your responses, @rajpawar02! We'll discuss them during our next team planning session. We'll circle back to you with updates after.

[workato-integration]

https://issues.newrelic.com/browse/NEWRELIC-3461

[askreet]

We are contemplating moving to NewRelic Logs and have hit a similar question in testing the default integration with newrelic_rpm. For us, we set DYNO to the name of the deployment for historical reasons (we have migrated from Heroku to Kubernetes), which is quite convenient because we now have hostname set as the deployment name.

However, we would also like the container name for specific troubleshooting situations. I'm surprised there doesn't appear to be a way to attach attributes to logs.

I'd be happy with any of the solutions above, such as putting the following in a Rails initializer:

# An environment variable we already provide to our containers, for example:
NewRelic::Agent.add_logging_attribute(pod_name: ENV.fetch('POD_NAME'))

[kaylareopelle]

Thanks for your interest, @askreet! It's great to hear about your use case (and see the ❤️ support so far from @joshbranham and @rajpawar02)! We're talking about this as a team and hope to have more information soon.

If anyone else comes across this post and is interested in this feature, please respond using a comment or an emoji. This feedback is a huge help for our team to understand interest for custom attributes in logs and prioritize this work.

[santanaguy]

In our case we are using a forwarder in older agents that adds the custom tag of environment. Then we have partitions per environment.

Example, we configure the forwarder to tail logfiles and all the messages from a certain logfile will be tagged as env:dev. This is useful because sometimes in a single machine we have multiple environments.

Our previous use case was using VMs, and we hoped to have the same behavior now that we are moving newer services to Docker. We don't want the full fledged infrastructure integration with Kubernetes, the .NET Docker agent along with the app in the container is sufficient.

Have we lost that ability now with the automatic forwarding?

[kaylareopelle]

Hi @santanaguy. Unfortunately, automatic forwarding in the Ruby agent does not have the ability to add custom attributes, such as environment, to log events at this time. This feature request is the perfect place to express your interest in this capability.

Some language agents for New Relic have already added this feature. If you have agents in multiple languages you may be able to add custom attributes to automatically forwarded log events using those agents today.

[santanaguy]

Our use case is with the .net agent, is this also not possible? Obter o Outlook para Android<https://aka.ms/AAb9ysg>

…

________________________________ From: kaylareopelle ***@***.***> Sent: Tuesday, November 29, 2022 6:41:42 PM To: newrelic/newrelic-ruby-agent ***@***.***> Cc: santanaguy ***@***.***>; Mention ***@***.***> Subject: Re: [newrelic/newrelic-ruby-agent] Adding custom attributes to the logs via the agent (Issue #1141) Hi @santanaguy<https://github.com/santanaguy>. Unfortunately, automatic forwarding in the Ruby agent does not have the ability to add custom attributes, such as environment, to log events at this time. This feature request is the perfect place to express your interest in this capability. Some language agents for New Relic have already added this feature. If you have agents in multiple languages you may be able to add custom attributes to automatically forwarded log events using those agents today. — Reply to this email directly, view it on GitHub<#1141 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA3L76HCIXEMWFTDJ7PUPNTWKZE6NANCNFSM5VHZ6W6A>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[kaylareopelle]

@santanaguy - the .NET agent team is currently working on adding support for custom attributes with logs.

[kaylareopelle]

@santanaguy - The .NET agent's 10.4.0 release includes custom attributes for logging.
https://github.com/newrelic/newrelic-dotnet-agent/releases/tag/v10.4.0

[santanaguy]

Hey, thanks for the heads up. I tried to enable the environment variable NEW_RELIC_APPLICATION_LOGGING_FORWARDING_CONTEXT_DATA_ENABLED but I did not get anything new on my logs. I am using the default Microsoft.Extensions.Logging library, not sure if I should be seeing something.

Another thing (not entirely related) is that the agent simply doesn't map correctly the INFORMATION logging level. Is this a limitation of using the Microsoft.Extensions.Logging lib?

[kaylareopelle]

Hi @santanaguy, glad to hear you're testing out the new instrumentation!

Unfortunately I don't know much about how the .NET agent works. Could you post your questions in a new issue in the .NET agent repository? https://github.com/newrelic/newrelic-dotnet-agent

The .NET agent engineers should be able to help you there!
cc @JcolemanNR

[angelatan2]

Hey team! Please add your planning poker estimate with Zenhub @fallwith @hannahramadan @kaylareopelle @tannalynn

[kaylareopelle]

Hi folks! We've added two new features to support custom attributes on New Relic log events:

• An API called NewRelic::Agent.add_custom_log_attributes: PR Create custom_log_attributes API #2084
• A configuration option named application_logging.forwarding.custom_attributes: PR Set custom log attributes using newrelic.yml or environment variables #2087

These features will be included in our next release. Thank you all for sharing your ideas and thoughts to design a solution!

cc: @rajpawar02 @askreet @santanaguy @joshbranham @erdzan12 @murkitsa

[kaylareopelle]

Version 9.3.0 of the agent, which includes this feature, has been released! 🎉