Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix lossiness when submitting timestamped custom metrics #527

Merged
merged 3 commits into from
Oct 31, 2024

Conversation

nhulston
Copy link
Contributor

@nhulston nhulston commented Oct 30, 2024

What does this PR do?

Fixes an issue in ThreadStatsWriter.flush() where constant_tags were accumulating duplicate tags over multiple Lambda invocations, leading to metric loss for metrics with timestamps. This code was the issue:

if tags:
    self.thread_stats.constant_tags = self.thread_stats.constant_tags + tags

Since self.thread_stats.constant_tags persists across multiple invocations within the same Lambda execution context, we kept appending duplicate tags after each invocation.

This issue only affected metrics with a timestamp, since they use ThreadStatsWriter for submission. After ~12 invocations, the payloads were too large, so custom metrics were being dropped.

This PR modifies the flush() method to still add the provided tags, but it now resets constant_tags to its original state afterward.

Motivation

#514

Testing Guidelines

Manual testing showed that metrics were no longer being dropped after these changes.

Added unit tests

Additional Notes

For people who are more familiar with this library, please make sure that these changes don't have any side effects that I did not consider.

Types of Changes

  • Bug fix
  • New feature
  • Breaking change
  • Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

  • This PR's description is comprehensive
  • This PR contains breaking changes that are documented in the description
  • This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
  • This PR impacts documentation, and it has been updated (or a ticket has been logged)
  • This PR's changes are covered by the automated tests
  • This PR collects user input/sensitive content into Datadog
  • This PR passes the integration tests (ask a Datadog member to run the tests)

@nhulston nhulston force-pushed the nicholas.hulston/fix-dropped-metrics branch from 063de03 to 8d4488c Compare October 30, 2024 18:51
@nhulston nhulston force-pushed the nicholas.hulston/fix-dropped-metrics branch from 8d4488c to d7c43d5 Compare October 30, 2024 20:37
@nhulston nhulston marked this pull request as ready for review October 30, 2024 20:56
@nhulston nhulston requested a review from a team as a code owner October 30, 2024 20:56
Copy link
Contributor

@purple4reina purple4reina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good find!

@nhulston nhulston merged commit 59e6e7c into main Oct 31, 2024
51 checks passed
@nhulston nhulston deleted the nicholas.hulston/fix-dropped-metrics branch October 31, 2024 13:15
@tianchu
Copy link
Collaborator

tianchu commented Nov 6, 2024

Great finding!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants