ekump/adding-test-agent-to-trace-utils

[ekump]

What does this PR do?

Introduces the test agent to libdatadog to be used for integration tests in trace-utils.

The test agent runs inside a docker container that is spun up/down within a given test via testcontainers-rs.

The DatadogTestAgent struct has been created as a wrapper around the testcontainers implementation. It also provides some helper functions to make it easier to write tests that leverage the test agent.

A single integration test has been added that performs a snapshot test via the test-agent for a trace.

Motivation

There is imminent work around trace_utils planned that could potentially modify trace payloads. These tests should provide some confidence for that work.

Additional Notes

The new tests naturally require docker. The README has been updated with instructions on how to skip these tests locally if you're not running docker. Alternatively, if people feel strongly we can invert this and default to not running these tests by default locally via a cargo feature.

The test will run on github CI only on Ubuntu for the time being.

How to test the change?

cargo nextest run

[pr-commenter]

Benchmarks

Comparison

Parameters

	Baseline	Candidate
config	baseline	candidate
git_branch	main	ekump/add-test-agent-to-trace-utils-2
git_commit_date	1720520743	1720531028
git_commit_sha	25f5c42	92c3fdd
iterations	748.0	787.0

See matching parameters

	Baseline	Candidate
ci_job_date	1720531380	1720531380
ci_job_id	566586950	566586950
ci_pipeline_id	38702639	38702639
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 0 metrics, 0 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:sql/obfuscate_sql_string	better [-3.248µs; -3.226µs] or [-4.867%; -4.835%]	63.493µs	66.730µs

Candidate

Candidate benchmark details

Group 1

iterations	config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
787.0	candidate	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720531380	566586950	38702639	92c3fdd	1720531028	ekump/add-test-agent-to-trace-utils-2

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sql/obfuscate_sql_string	execution_time	63.390µs	63.493µs ± 0.046µs	63.489µs ± 0.032µs	63.527µs	63.572µs	63.582µs	63.590µs	0.16%	-0.003	-0.637	0.07%	0.005µs	1	100

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sql/obfuscate_sql_string	execution_time	[63.484µs; 63.502µs] or [-0.014%; +0.014%]	None	None	None

Warnings

Scenario: sql/obfuscate_sql_string, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: sql/obfuscate_sql_string, Metric: execution_time

Sample size is 100, which is lower than 105.

The minimal sample size in case of normal distribution to achieve significance
level of 0.05 for difference of means with effect size Cohen's d = 0.5 must be at
least 105.

The conclusions from confidence intervals may be invalid.
---------------------------------------------------------------------------

Baseline

Baseline benchmark details

Group 1

iterations	config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
748.0	baseline	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720531380	566586950	38702639	25f5c42	1720520743	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sql/obfuscate_sql_string	execution_time	66.639µs	66.730µs ± 0.030µs	66.736µs ± 0.018µs	66.751µs	66.774µs	66.784µs	66.786µs	0.08%	-0.635	0.503	0.04%	0.003µs	1	100

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sql/obfuscate_sql_string	execution_time	[66.724µs; 66.736µs] or [-0.009%; +0.009%]	None	None	None

Warnings

Scenario: sql/obfuscate_sql_string, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: sql/obfuscate_sql_string, Metric: execution_time

Sample size is 100, which is lower than 105.

The minimal sample size in case of normal distribution to achieve significance
level of 0.05 for difference of means with effect size Cohen's d = 0.5 must be at
least 105.

The conclusions from confidence intervals may be invalid.
---------------------------------------------------------------------------

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 96.80851% with 3 lines in your changes missing coverage. Please review.

Project coverage is 70.30%. Comparing base (25f5c42) to head (92c3fdd).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #516      +/-   ##
==========================================
+ Coverage   70.21%   70.30%   +0.08%     
==========================================
  Files         205      206       +1     
  Lines       27712    27806      +94     
==========================================
+ Hits        19458    19549      +91     
- Misses       8254     8257       +3     

Components	Coverage Δ
crashtracker	16.86% <ø> (ø)
datadog-alloc	98.73% <ø> (ø)
data-pipeline	51.15% <ø> (ø)
data-pipeline-ffi	0.00% <ø> (ø)
ddcommon	86.68% <ø> (ø)
ddcommon-ffi	75.31% <ø> (ø)
ddtelemetry	59.02% <ø> (ø)
ipc	84.13% <ø> (ø)
profiling	78.68% <ø> (ø)
profiling-ffi	58.26% <ø> (ø)
serverless	0.00% <ø> (ø)
sidecar	35.69% <ø> (ø)
sidecar-ffi	0.00% <ø> (ø)
spawn-worker	54.98% <ø> (ø)
trace-mini-agent	70.93% <ø> (ø)
trace-normalization	98.24% <ø> (ø)
trace-obfuscation	95.75% <ø> (ø)
trace-protobuf	77.16% <ø> (ø)
trace-utils	90.68% <96.80%> (+0.25%)	⬆️

[bantonsson]

I, for one, welcome our new docker running overlords...

[VianneyRuhlmann]

I'm impressed by how smooth it is to run test with docker within cargo 🤩

[VianneyRuhlmann]

Should this use TEST_AGENT_PORT ?

[ekump]

yes, good catch.