Design RFC for crashtracker

[danielsn]

What does this PR do?

Adds an RFC laying out the design considerations, risks, and mitigations for the crashtracker.

Motivation

We should clearly document this as the tool evolves.

Additional Notes

Anything else we should know when reviewing?

How to test the change?

This is documentation. No tests

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.73%. Comparing base (214f046) to head (eb35127).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #696      +/-   ##
==========================================
- Coverage   71.73%   71.73%   -0.01%     
==========================================
  Files         271      271              
  Lines       41143    41143              
==========================================
- Hits        29514    29513       -1     
- Misses      11629    11630       +1     

Components	Coverage Δ
crashtracker	20.15% <ø> (+0.05%)	⬆️
crashtracker-ffi	8.97% <ø> (ø)
datadog-alloc	98.73% <ø> (ø)
data-pipeline	91.86% <ø> (-0.07%)	⬇️
data-pipeline-ffi	0.00% <ø> (ø)
ddcommon	82.74% <ø> (ø)
ddcommon-ffi	69.12% <ø> (ø)
ddtelemetry	59.10% <ø> (ø)
ddtelemetry-ffi	22.13% <ø> (ø)
dogstatsd	89.22% <ø> (ø)
dogstatsd-client	79.77% <ø> (ø)
ipc	82.65% <ø> (ø)
profiling	84.30% <ø> (ø)
profiling-ffi	77.46% <ø> (ø)
serverless	0.00% <ø> (ø)
sidecar	36.13% <ø> (ø)
sidecar-ffi	0.00% <ø> (ø)
spawn-worker	50.36% <ø> (ø)
tinybytes	94.77% <ø> (ø)
trace-mini-agent	72.45% <ø> (ø)
trace-normalization	98.25% <ø> (ø)
trace-obfuscation	95.77% <ø> (ø)
trace-protobuf	77.67% <ø> (ø)
trace-utils	93.14% <ø> (ø)

[pr-commenter]

Benchmarks

Comparison

Benchmark execution time: 2024-10-31 18:56:47

Comparing candidate commit eb35127 in PR branch dsn/crashtracker-design-rfc with baseline commit 214f046 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 2 unstable metrics.

Candidate

Candidate benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	eb35127	1730400376	dsn/crashtracker-design-rfc

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching string interning on wordpress profile	execution_time	142.878µs	143.953µs ± 0.606µs	143.886µs ± 0.159µs	144.058µs	144.494µs	145.629µs	150.916µs	4.89%	7.860	86.185	0.42%	0.043µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching string interning on wordpress profile	execution_time	[143.869µs; 144.037µs] or [-0.058%; +0.058%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	eb35127	1730400376	dsn/crashtracker-design-rfc

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_trace/test_trace	execution_time	257.591ns	269.617ns ± 15.704ns	261.573ns ± 2.896ns	276.730ns	299.699ns	310.975ns	312.680ns	19.54%	1.313	0.360	5.81%	1.110ns	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_trace/test_trace	execution_time	[267.441ns; 271.793ns] or [-0.807%; +0.807%]	None	None	None

Group 3

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	eb35127	1730400376	dsn/crashtracker-design-rfc

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	187.391µs	191.931µs ± 1.326µs	192.129µs ± 0.957µs	192.900µs	193.896µs	194.246µs	194.660µs	1.32%	-0.435	-0.112	0.69%	0.094µs	1	200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	throughput	5137151.120op/s	5210445.458op/s ± 36122.501op/s	5204837.447op/s ± 25919.411op/s	5235116.146op/s	5269703.313op/s	5299856.883op/s	5336431.085op/s	2.53%	0.471	-0.045	0.69%	2554.247op/s	1	200
normalization/normalize_name/normalize_name/bad-name	execution_time	18.177µs	18.208µs ± 0.029µs	18.213µs ± 0.024µs	18.226µs	18.249µs	18.303µs	18.378µs	0.91%	1.654	5.982	0.16%	0.002µs	1	200
normalization/normalize_name/normalize_name/bad-name	throughput	54413035.519op/s	54919979.937op/s ± 87193.722op/s	54905836.632op/s ± 72113.963op/s	54994919.378op/s	55009022.529op/s	55013534.192op/s	55014501.916op/s	0.20%	-1.629	5.812	0.16%	6165.527op/s	1	200
normalization/normalize_name/normalize_name/good	execution_time	11.228µs	11.256µs ± 0.021µs	11.249µs ± 0.011µs	11.276µs	11.295µs	11.309µs	11.325µs	0.68%	0.852	-0.138	0.18%	0.001µs	1	200
normalization/normalize_name/normalize_name/good	throughput	88299367.970op/s	88838903.804op/s ± 162955.592op/s	88899847.030op/s ± 85720.977op/s	88957444.810op/s	89023976.592op/s	89056342.645op/s	89063924.240op/s	0.18%	-0.846	-0.157	0.18%	11522.700op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	[191.747µs; 192.115µs] or [-0.096%; +0.096%]	None	None	None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	throughput	[5205439.227op/s; 5215451.689op/s] or [-0.096%; +0.096%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	execution_time	[18.204µs; 18.212µs] or [-0.022%; +0.022%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	throughput	[54907895.726op/s; 54932064.148op/s] or [-0.022%; +0.022%]	None	None	None
normalization/normalize_name/normalize_name/good	execution_time	[11.254µs; 11.259µs] or [-0.025%; +0.025%]	None	None	None
normalization/normalize_name/normalize_name/good	throughput	[88816319.726op/s; 88861487.881op/s] or [-0.025%; +0.025%]	None	None	None

Group 4

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	eb35127	1730400376	dsn/crashtracker-design-rfc

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	622.363µs	624.274µs ± 0.550µs	624.332µs ± 0.391µs	624.691µs	625.030µs	625.402µs	625.588µs	0.20%	-0.517	0.260	0.09%	0.039µs	1	200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	throughput	1598495.040op/s	1601861.347op/s ± 1411.390op/s	1601711.910op/s ± 1003.929op/s	1602752.934op/s	1604355.753op/s	1605552.256op/s	1606779.508op/s	0.32%	0.522	0.268	0.09%	99.800op/s	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	461.280µs	462.189µs ± 0.475µs	462.120µs ± 0.314µs	462.486µs	463.021µs	463.451µs	463.551µs	0.31%	0.531	-0.038	0.10%	0.034µs	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	throughput	2157260.425op/s	2163619.622op/s ± 2224.268op/s	2163940.966op/s ± 1468.382op/s	2165168.944op/s	2166812.802op/s	2167696.697op/s	2167878.948op/s	0.18%	-0.525	-0.045	0.10%	157.279op/s	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	178.179µs	178.371µs ± 0.107µs	178.369µs ± 0.045µs	178.401µs	178.577µs	178.767µs	178.851µs	0.27%	1.476	4.126	0.06%	0.008µs	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	throughput	5591241.007op/s	5606287.162op/s ± 3354.719op/s	5606361.497op/s ± 1417.499op/s	5608410.838op/s	5610837.664op/s	5611716.389op/s	5612347.493op/s	0.11%	-1.469	4.097	0.06%	237.214op/s	1	200
normalization/normalize_service/normalize_service/[empty string]	execution_time	47.140µs	47.314µs ± 0.075µs	47.306µs ± 0.046µs	47.359µs	47.434µs	47.484µs	47.630µs	0.69%	0.610	1.083	0.16%	0.005µs	1	200
normalization/normalize_service/normalize_service/[empty string]	throughput	20995002.296op/s	21135535.039op/s ± 33467.443op/s	21139078.497op/s ± 20569.331op/s	21156756.147op/s	21187168.120op/s	21195735.718op/s	21213339.851op/s	0.35%	-0.597	1.047	0.16%	2366.506op/s	1	200
normalization/normalize_service/normalize_service/test_ASCII	execution_time	48.933µs	49.450µs ± 0.319µs	49.330µs ± 0.165µs	49.691µs	50.066µs	50.197µs	50.246µs	1.86%	0.769	-0.603	0.64%	0.023µs	1	200
normalization/normalize_service/normalize_service/test_ASCII	throughput	19902032.803op/s	20223153.170op/s ± 129897.864op/s	20271803.891op/s ± 67901.191op/s	20326635.326op/s	20366885.123op/s	20391365.206op/s	20436040.818op/s	0.81%	-0.753	-0.630	0.64%	9185.166op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	[624.198µs; 624.350µs] or [-0.012%; +0.012%]	None	None	None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	throughput	[1601665.742op/s; 1602056.952op/s] or [-0.012%; +0.012%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	[462.123µs; 462.255µs] or [-0.014%; +0.014%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	throughput	[2163311.360op/s; 2163927.884op/s] or [-0.014%; +0.014%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	[178.356µs; 178.386µs] or [-0.008%; +0.008%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	throughput	[5605822.230op/s; 5606752.094op/s] or [-0.008%; +0.008%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	execution_time	[47.303µs; 47.324µs] or [-0.022%; +0.022%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	throughput	[21130896.773op/s; 21140173.305op/s] or [-0.022%; +0.022%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	execution_time	[49.406µs; 49.495µs] or [-0.089%; +0.089%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	throughput	[20205150.576op/s; 20241155.765op/s] or [-0.089%; +0.089%]	None	None	None

Group 5

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	eb35127	1730400376	dsn/crashtracker-design-rfc

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
tags/replace_trace_tags	execution_time	2.657µs	2.681µs ± 0.016µs	2.677µs ± 0.005µs	2.682µs	2.731µs	2.735µs	2.736µs	2.19%	2.298	4.974	0.60%	0.001µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
tags/replace_trace_tags	execution_time	[2.678µs; 2.683µs] or [-0.083%; +0.083%]	None	None	None

Group 6

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	eb35127	1730400376	dsn/crashtracker-design-rfc

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
concentrator/add_spans_to_concentrator	execution_time	9.072ms	9.105ms ± 0.012ms	9.105ms ± 0.008ms	9.112ms	9.125ms	9.131ms	9.145ms	0.44%	0.114	0.402	0.13%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
concentrator/add_spans_to_concentrator	execution_time	[9.103ms; 9.106ms] or [-0.019%; +0.019%]	None	None	None

Group 7

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	eb35127	1730400376	dsn/crashtracker-design-rfc

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching deserializing traces from msgpack to their internal representation	execution_time	1.120µs	1.179µs ± 0.023µs	1.183µs ± 0.015µs	1.197µs	1.201µs	1.204µs	1.205µs	1.87%	-1.079	0.417	1.92%	0.002µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching deserializing traces from msgpack to their internal representation	execution_time	[1.175µs; 1.182µs] or [-0.267%; +0.267%]	None	None	None

Group 8

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	eb35127	1730400376	dsn/crashtracker-design-rfc

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
redis/obfuscate_redis_string	execution_time	37.957µs	38.641µs ± 1.159µs	38.108µs ± 0.104µs	38.306µs	41.120µs	41.206µs	41.867µs	9.86%	1.684	0.918	2.99%	0.082µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
redis/obfuscate_redis_string	execution_time	[38.480µs; 38.801µs] or [-0.416%; +0.416%]	None	None	None

Group 9

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	eb35127	1730400376	dsn/crashtracker-design-rfc

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
credit_card/is_card_number/	execution_time	4.606µs	4.615µs ± 0.002µs	4.614µs ± 0.001µs	4.616µs	4.618µs	4.619µs	4.624µs	0.21%	0.392	1.814	0.05%	0.000µs	1	200
credit_card/is_card_number/	throughput	216261465.001op/s	216707850.904op/s ± 103398.903op/s	216718507.762op/s ± 62413.120op/s	216777047.982op/s	216850578.875op/s	216888369.720op/s	217114580.291op/s	0.18%	-0.387	1.810	0.05%	7311.407op/s	1	200
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	90.336µs	91.318µs ± 0.586µs	91.220µs ± 0.267µs	91.539µs	92.169µs	93.074µs	96.057µs	5.30%	3.091	20.539	0.64%	0.041µs	1	200
credit_card/is_card_number/ 3782-8224-6310-005	throughput	10410538.645op/s	10951140.555op/s ± 68970.669op/s	10962496.220op/s ± 32177.965op/s	10990067.454op/s	11029958.141op/s	11059552.019op/s	11069806.198op/s	0.98%	-2.854	18.090	0.63%	4876.963op/s	1	200
credit_card/is_card_number/ 378282246310005	execution_time	82.892µs	83.836µs ± 0.579µs	83.758µs ± 0.324µs	84.091µs	84.692µs	85.226µs	87.898µs	4.94%	2.120	11.217	0.69%	0.041µs	1	200
credit_card/is_card_number/ 378282246310005	throughput	11376823.136op/s	11928572.084op/s ± 81222.188op/s	11939205.574op/s ± 46010.028op/s	11981945.914op/s	12031924.501op/s	12047916.524op/s	12063909.768op/s	1.04%	-1.949	9.757	0.68%	5743.276op/s	1	200
credit_card/is_card_number/37828224631	execution_time	4.599µs	4.615µs ± 0.003µs	4.614µs ± 0.002µs	4.616µs	4.618µs	4.620µs	4.631µs	0.35%	0.239	10.822	0.06%	0.000µs	1	200
credit_card/is_card_number/37828224631	throughput	215945959.897op/s	216704734.020op/s ± 126013.180op/s	216708581.664op/s ± 75867.785op/s	216784979.713op/s	216857388.588op/s	216898758.875op/s	217448653.971op/s	0.34%	-0.217	10.818	0.06%	8910.477op/s	1	200
credit_card/is_card_number/378282246310005	execution_time	79.498µs	80.503µs ± 0.401µs	80.508µs ± 0.265µs	80.770µs	81.174µs	81.422µs	81.497µs	1.23%	0.061	-0.207	0.50%	0.028µs	1	200
credit_card/is_card_number/378282246310005	throughput	12270331.301op/s	12422130.020op/s ± 61829.174op/s	12421175.268op/s ± 40873.384op/s	12463782.211op/s	12518222.032op/s	12573078.414op/s	12578903.689op/s	1.27%	-0.034	-0.207	0.50%	4371.983op/s	1	200
credit_card/is_card_number/37828224631000521389798	execution_time	59.264µs	59.526µs ± 0.203µs	59.522µs ± 0.167µs	59.640µs	59.870µs	60.060µs	60.092µs	0.96%	0.512	-0.485	0.34%	0.014µs	1	200
credit_card/is_card_number/37828224631000521389798	throughput	16641155.152op/s	16799711.145op/s ± 57165.501op/s	16800634.341op/s ± 47351.850op/s	16856908.281op/s	16870704.191op/s	16872475.981op/s	16873515.330op/s	0.43%	-0.499	-0.511	0.34%	4042.211op/s	1	200
credit_card/is_card_number/x371413321323331	execution_time	7.218µs	7.223µs ± 0.006µs	7.223µs ± 0.002µs	7.224µs	7.226µs	7.228µs	7.296µs	1.01%	11.082	141.794	0.08%	0.000µs	1	200
credit_card/is_card_number/x371413321323331	throughput	137065784.748op/s	138448087.279op/s ± 106395.816op/s	138450130.269op/s ± 29625.165op/s	138489270.544op/s	138518501.917op/s	138524924.520op/s	138536213.366op/s	0.06%	-11.033	140.952	0.08%	7523.320op/s	1	200
credit_card/is_card_number_no_luhn/	execution_time	4.602µs	4.614µs ± 0.002µs	4.614µs ± 0.001µs	4.616µs	4.618µs	4.620µs	4.628µs	0.30%	0.485	8.429	0.05%	0.000µs	1	200
credit_card/is_card_number_no_luhn/	throughput	216063778.598op/s	216712404.188op/s ± 111197.925op/s	216714975.005op/s ± 66621.588op/s	216781780.927op/s	216854180.070op/s	216873771.194op/s	217308855.953op/s	0.27%	-0.470	8.413	0.05%	7862.881op/s	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	73.956µs	74.422µs ± 0.171µs	74.437µs ± 0.083µs	74.508µs	74.660µs	74.739µs	75.560µs	1.51%	1.178	8.884	0.23%	0.012µs	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	13234518.212op/s	13436932.236op/s ± 30814.861op/s	13434121.087op/s ± 15034.139op/s	13454077.199op/s	13483526.976op/s	13503869.719op/s	13521635.970op/s	0.65%	-1.114	8.448	0.23%	2178.940op/s	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	66.327µs	66.838µs ± 0.249µs	66.809µs ± 0.174µs	67.010µs	67.280µs	67.440µs	67.637µs	1.24%	0.453	-0.243	0.37%	0.018µs	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	14784718.239op/s	14961825.897op/s ± 55745.107op/s	14968031.315op/s ± 39160.882op/s	15004219.203op/s	15040398.220op/s	15045180.919op/s	15076910.335op/s	0.73%	-0.436	-0.272	0.37%	3941.774op/s	1	200
credit_card/is_card_number_no_luhn/37828224631	execution_time	4.578µs	4.614µs ± 0.003µs	4.615µs ± 0.001µs	4.616µs	4.618µs	4.619µs	4.619µs	0.11%	-7.744	88.433	0.07%	0.000µs	1	200
credit_card/is_card_number_no_luhn/37828224631	throughput	216478261.317op/s	216712068.717op/s ± 149281.700op/s	216706046.108op/s ± 59925.998op/s	216768889.153op/s	216840753.502op/s	216871094.166op/s	218446743.853op/s	0.80%	7.807	89.373	0.07%	10555.810op/s	1	200
credit_card/is_card_number_no_luhn/378282246310005	execution_time	62.350µs	62.652µs ± 0.172µs	62.615µs ± 0.107µs	62.751µs	62.987µs	63.136µs	63.219µs	0.97%	0.896	0.299	0.27%	0.012µs	1	200
credit_card/is_card_number_no_luhn/378282246310005	throughput	15817955.891op/s	15961397.176op/s ± 43721.111op/s	15970645.955op/s ± 27256.494op/s	15995560.832op/s	16008317.763op/s	16034342.713op/s	16038439.328op/s	0.42%	-0.884	0.270	0.27%	3091.549op/s	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	59.261µs	59.578µs ± 0.230µs	59.589µs ± 0.179µs	59.743µs	59.958µs	60.100µs	60.106µs	0.87%	0.346	-0.778	0.39%	0.016µs	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	16637249.581op/s	16785050.559op/s ± 64709.375op/s	16781744.084op/s ± 50306.088op/s	16850095.102op/s	16871160.186op/s	16872798.568op/s	16874551.489op/s	0.55%	-0.333	-0.794	0.38%	4575.644op/s	1	200
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	7.218µs	7.222µs ± 0.003µs	7.222µs ± 0.002µs	7.224µs	7.227µs	7.229µs	7.238µs	0.22%	1.380	3.902	0.04%	0.000µs	1	200
credit_card/is_card_number_no_luhn/x371413321323331	throughput	138161876.440op/s	138457882.966op/s ± 54428.719op/s	138469889.775op/s ± 36910.909op/s	138498898.934op/s	138522865.277op/s	138534142.017op/s	138537917.828op/s	0.05%	-1.375	3.873	0.04%	3848.692op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
credit_card/is_card_number/	execution_time	[4.614µs; 4.615µs] or [-0.007%; +0.007%]	None	None	None
credit_card/is_card_number/	throughput	[216693520.810op/s; 216722180.997op/s] or [-0.007%; +0.007%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	[91.237µs; 91.400µs] or [-0.089%; +0.089%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	throughput	[10941581.884op/s; 10960699.226op/s] or [-0.087%; +0.087%]	None	None	None
credit_card/is_card_number/ 378282246310005	execution_time	[83.756µs; 83.916µs] or [-0.096%; +0.096%]	None	None	None
credit_card/is_card_number/ 378282246310005	throughput	[11917315.469op/s; 11939828.698op/s] or [-0.094%; +0.094%]	None	None	None
credit_card/is_card_number/37828224631	execution_time	[4.614µs; 4.615µs] or [-0.008%; +0.008%]	None	None	None
credit_card/is_card_number/37828224631	throughput	[216687269.805op/s; 216722198.235op/s] or [-0.008%; +0.008%]	None	None	None
credit_card/is_card_number/378282246310005	execution_time	[80.448µs; 80.559µs] or [-0.069%; +0.069%]	None	None	None
credit_card/is_card_number/378282246310005	throughput	[12413561.091op/s; 12430698.949op/s] or [-0.069%; +0.069%]	None	None	None
credit_card/is_card_number/37828224631000521389798	execution_time	[59.497µs; 59.554µs] or [-0.047%; +0.047%]	None	None	None
credit_card/is_card_number/37828224631000521389798	throughput	[16791788.557op/s; 16807633.734op/s] or [-0.047%; +0.047%]	None	None	None
credit_card/is_card_number/x371413321323331	execution_time	[7.222µs; 7.224µs] or [-0.011%; +0.011%]	None	None	None
credit_card/is_card_number/x371413321323331	throughput	[138433341.842op/s; 138462832.716op/s] or [-0.011%; +0.011%]	None	None	None
credit_card/is_card_number_no_luhn/	execution_time	[4.614µs; 4.615µs] or [-0.007%; +0.007%]	None	None	None
credit_card/is_card_number_no_luhn/	throughput	[216696993.225op/s; 216727815.151op/s] or [-0.007%; +0.007%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	[74.398µs; 74.446µs] or [-0.032%; +0.032%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	[13432661.593op/s; 13441202.879op/s] or [-0.032%; +0.032%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	[66.803µs; 66.872µs] or [-0.052%; +0.052%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	[14954100.161op/s; 14969551.633op/s] or [-0.052%; +0.052%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	execution_time	[4.614µs; 4.615µs] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	throughput	[216691379.709op/s; 216732757.725op/s] or [-0.010%; +0.010%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	execution_time	[62.628µs; 62.675µs] or [-0.038%; +0.038%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	throughput	[15955337.850op/s; 15967456.501op/s] or [-0.038%; +0.038%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	[59.546µs; 59.610µs] or [-0.053%; +0.053%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	[16776082.462op/s; 16794018.656op/s] or [-0.053%; +0.053%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	[7.222µs; 7.223µs] or [-0.005%; +0.005%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	throughput	[138450339.669op/s; 138465426.263op/s] or [-0.005%; +0.005%]	None	None	None

Group 10

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	eb35127	1730400376	dsn/crashtracker-design-rfc

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
write only interface	execution_time	1.341µs	3.192µs ± 1.389µs	3.044µs ± 0.024µs	3.067µs	3.114µs	13.830µs	14.881µs	388.92%	7.585	57.745	43.41%	0.098µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
write only interface	execution_time	[2.999µs; 3.385µs] or [-6.032%; +6.032%]	None	None	None

Group 11

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	eb35127	1730400376	dsn/crashtracker-design-rfc

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
two way interface	execution_time	18.201µs	24.258µs ± 10.502µs	19.455µs ± 0.247µs	20.121µs	39.181µs	41.501µs	98.196µs	404.74%	3.451	18.088	43.18%	0.743µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
two way interface	execution_time	[22.802µs; 25.713µs] or [-6.000%; +6.000%]	None	None	None

Group 12

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	eb35127	1730400376	dsn/crashtracker-design-rfc

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sql/obfuscate_sql_string	execution_time	72.084µs	72.202µs ± 0.132µs	72.179µs ± 0.026µs	72.206µs	72.335µs	72.535µs	73.662µs	2.05%	7.674	76.353	0.18%	0.009µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sql/obfuscate_sql_string	execution_time	[72.183µs; 72.220µs] or [-0.025%; +0.025%]	None	None	None

Baseline

Omitted due to size.

[sanchda]

Left some comments, mostly speculative around future design. I do not think my comments should necessarily change the RFC as it is written, I'm just adding them here as a record for future discussions.

[ivoanjo]

Thanks for putting this together! I think this RFC will be a very useful resource to make sure everyone is on the same page when discussing the crashtracker.

I've left a bunch of suggestions, but I think overall this is pretty good 👍