metrics.md

Benchmark Characterization

Code Size Statistics

The following metrics characterize the size of the benchmarks. The numbers are based on the SOMns benchmark implementations, but should be similar to the numbers one would measure for the other languages.

Executed Lines of Code

Compared to the classic notion of lines of code (LOC), we count only lines of code that have been executed at least once to measure the dynamic size of the program instead of its static size.

Classes

The number of classes includes only classes of which at least one method was executed. It is stable across languages with the exception of JavaScript which does not use classes in the version of the language used for this work.

Executed Methods

Similar to counting executed LOC, we count methods that have been executed at least once.

Per Iteration Methods

In addition to executed methods, we further distinguish methods that are executed for each benchmark iteration. Thus, we separate out code that was only executed once during startup or shutdown. The number of per iteration methods indicates the methods that get likely compiled during benchmarking.

	Executed Lines	Classes	Executed Methods	Per Iteration Methods
CD	356	16	43	41
DeltaBlue	387	20	99	75
Havlak	421	18	110	87
Json	232	14	56	56
Richards	279	12	47	47
Bounce	42	5	11	11
List	30	2	9	9
Mandelbrot	39	0	2	2
NBody	105	3	14	14
Permute	33	3	13	13
Queens	36	3	13	13
Sieve	22	3	9	9
Storage	23	4	10	10
Towers	42	2	12	12

Dynamic Metrics

The dynamic metrics characterize the behavior of the benchmarks in more detail.

Method Calls

For method calls, we count at each call site, i.e., the lexical point in the program where a call is made, the number of different receiver types, as well as the number of different target methods that were activated at that point. The number of target methods should always be smaller or equal to the number of different receiver types.

Operators such as + or * are excluded from the method call count.

Observed receiver polymorphism, i.e., different observed receiver types/classes:

	Num.Rcvrs	Call Sites	Calls
CD	1	166	75,892,212
	2	3	23,151,272
DeltaBlue	1	171	6,219,974
	2	8	480,130
	3	8	3,576,416
	4	7	840,146
Havlak	1	256	79,581,122
	2	3	2,720,556
Json	1	163	32,115,418
	5	2	678,200
Richards	1	108	95,521,018
	2	2	10,000
Bounce	1	31	17,736,018
List	1	28	17,271,018
Mandelbrot	1	10	16
NBody	1	38	8,500,102
Permute	1	25	51,982,018
Queens	1	28	29,088,018
Sieve	1	19	60,018
Storage	1	23	32,774,018
Towers	1	27	39,390,018

Observed target polymorphism, i.e., different activated methods:

	Num.Targets	Call Sites	Calls
CD	1	166	75,892,212
	2	3	23,151,272
DeltaBlue	1	180	6,892,136
	2	10	936,150
	3	4	3,288,380
Havlak	1	257	80,439,690
	2	2	1,861,988
Json	1	165	32,793,618
Richards	1	110	95,531,018
Bounce	1	31	17,736,018
List	1	28	17,271,018
Mandelbrot	1	10	16
NBody	1	38	8,500,102
Permute	1	25	51,982,018
Queens	1	28	29,088,018
Sieve	1	19	60,018
Storage	1	23	32,774,018
Towers	1	27	39,390,018

Closure Applications

Similar to method calls, we measure the number of lexical closures observed at a closure application site. A closure application site is the lexical point where a closure is executed.

	Num.Targets	Sites	Activations
CD	1	2	1,345,202
	2	1	677,440
	3	1	872,694
	8	1	2,638,378
DeltaBlue	1	13	336,022
	2	1	82
	3	1	447,442
	8	2	7,984,628
Havlak	1	1	531,748
	2	1	1,701,912
	3	1	2,664,704
	6	1	1,324
	9	2	13,571,016
Json	2	3	2,070,004
Richards	1	1	7,600
	2	1	202
	3	1	1,869,402
	4	1	13,158,000
Bounce	1	2	15,300,000
	2	2	15,303,002
	3	1	15,453,002
List	1	1	3,002
	2	1	3,002
Mandelbrot	1	2	4
NBody	1	2	2,500,012
	2	1	2,500,010
	7	1	10,500,042
Permute	1	2	10,090,000
	2	2	16,004
Queens	1	1	960,000
	2	1	2,002
	4	1	18,502,002
Sieve	1	1	30,000,000
	2	1	6,002
	3	1	60,000,002
Storage	1	2	21,840,000
	2	2	10,924,004
Towers	1	2	18,002
	2	1	1,202

Maximum Stack Height

As an indication for the recursive behavior, we measure the maximal observed stack height, i.e., the number of method activations on the runtime stack.

	Max Stack Height
CD	35
DeltaBlue	37
Havlak	1717
Json	43
Richards	31
Bounce	30
List	37
Mandelbrot	15
NBody	24
Permute	44
Queens	56
Sieve	26
Storage	75
Towers	34

Loops

We count loops that have been activated at least once. Furthermore, we count the number of times a loop body has been executed.

	Loops	Iterations
CD	7	2,314,044
DeltaBlue	6	348,403
Havlak	8	4,397,585
Json	6	1,220,001
Richards	3	1,246,401
Bounce	1	78,001
List	2	4,213,501
Mandelbrot	3	250,501
NBody	1	1,750,008
Permute	2	3,621,001
Queens	1	1,171,001
Sieve	2	2,013,001
Storage	1	1,365,001
Towers	2	601

Branches

We count the number of control flow branches that have been taken at least once. This includes if-branches, but also operations that have control flow semantics such as short-cutting or and and operators where the right-hand expression is executed conditionally. Furthermore, we count how often each branch is taken. The reported branch bias ratio is calculated with max(#true, #false) / (#true + #false).

	Branches	Activations	Bias
CD	96	106,182,895	0.81
DeltaBlue	65	6,824,975	0.87
Havlak	64	26,401,446	0.89
Json	69	19,359,103	0.89
Richards	37	102,788,003	0.76
Bounce	12	45,306,003	0.95
List	12	63,699,003	0.95
Mandelbrot	13	17,360,098	0.99
NBody	7	7,000,031	0.96
Permute	9	17,328,003	0.87
Queens	10	21,353,003	0.97
Sieve	9	80,220,003	0.98
Storage	8	12,288,003	0.94
Towers	11	14,762,403	0.93

Control Flow Statistics Overview

As an overview of the control flow metrics, the following plot aggregates these metrics and puts them into relation to each other.

Allocations

We track the number of arrays created as well as their overall size. Furthermore, we track the number of objects created and the number of declared fields.

Array allocations:

	Allocation Sites	#Arrays	Total Length
CD	2	142,230	8,331,250
DeltaBlue	5	108,214	645,645
Havlak	12	1,639,424	68,619,384
Json	3	102,600	4,867,400
Richards	1	900	3,800
Bounce	1	1,500	150,000
List	0	0	0
Mandelbrot	0	0	0
NBody	1	1	5
Permute	1	1,000	6,000
Queens	1	40,000	480,000
Sieve	1	3,000	15,000,000
Storage	2	5,461,000	27,880,000
Towers	1	600	1,800

Object allocations:

	Allocation Sites	#Objects	Total Requested Slots
CD	23	5386959	14462908
DeltaBlue	22	180051	696158
Havlak	23	2692930	8234883
Json	18	379800	585200
Richards	17	2700	12900
Bounce	11	151500	601500
List	10	46500	93000
Mandelbrot	9	0	0
NBody	11	6	36
Permute	9	0	0
Queens	9	0	0
Sieve	9	0	0
Storage	10	1000	1000
Towers	10	8400	16800

Object Field Accesses

We count object field reads and writes that were executed at least once. Furthermore, we report the number of accesses per iteration.

	Read Sites	Write Sites	Read Sites (per iter)	Write Sites (per iter)	Reads	Writes	Read Ratio
CD	233	35	229	35	140,991,969	6,273,733	0.96
DeltaBlue	206	40	192	39	20,072,860	2,772,733	0.88
Havlak	198	37	195	37	57,408,095	4,648,016	0.93
Json	93	9	89	9	35,697,206	8,677,601	0.80
Richards	138	46	135	46	217,152,006	59,662,901	0.78
Bounce	25	12	22	12	63,342,006	19,878,001	0.76
List	16	2	13	2	57,337,506	46,501	1.00
Mandelbrot	9	1	6	1	6	1	0.86
NBody	71	13	68	13	58,000,196	18,750,004	0.76
Permute	16	4	13	4	48,974,006	8,662,001	0.85
Queens	20	5	17	5	33,990,006	40,001	1.00
Sieve	10	1	7	1	3,006	1	1.00
Storage	16	4	13	4	19,116,006	9,558,001	0.67
Towers	19	6	16	6	39,198,006	14,753,401	0.73

Array Accesses

We count the sites of array reads and writes that were executed at least once. Furthermore, we count the number of array reads and writes per iteration.

	Read Sites	Write Sites	Read Sites (per iter)	Write Sites (per iter)	Reads	Writes	Read Ratio
CD	3	2	3	2	1,417,988	1,299,498	0.522
DeltaBlue	11	5	9	4	2,324,180	571,777	0.803
Havlak	23	20	23	20	10,182,376	5,814,705	0.637
Json	4	5	4	5	15,900	976,100	0.016
Richards	2	4	2	4	4,254,500	935,200	0.820
Bounce	1	1	1	1	7,500,000	150,000	0.980
List	0	0	0	0	0	0	0.000
Mandelbrot	0	0	0	0	0	0	0.000
NBody	6	5	6	5	5,000,021	5	1.000
Permute	2	3	2	3	20,156,000	20,162,000	0.500
Queens	3	6	3	6	26,280,000	8,150,000	0.763
Sieve	1	2	1	2	14,997,000	48,207,000	0.237
Storage	0	1	0	1	0	5,460,000	0.000
Towers	2	2	2	2	9,837,600	9,837,600	0.500

Allocation and Access Overview

As an overview of allocation and access metrics, the following plot aggregates and relates them to each other.

Variable Accesses

We report the number of sites of variable reads and writes that were executed at least once. This includes variables in methods and closures. Furthermore, we report the number of accesses per iteration.

	Read Sites	Write Sites	Read Sites (per iter)	Write Sites (per iter)	Reads	Writes	Read Ratio
CD	951	108	741	101	404,216,779	67,134,708	0.86
DeltaBlue	958	74	615	59	48,705,206	3,536,888	0.93
Havlak	1040	74	836	67	234,331,616	18,823,992	0.93
Json	574	24	354	17	82,042,539	1,257,405	0.98
Richards	662	42	459	35	369,377,939	33,872,605	0.92
Bounce	296	24	113	17	204,171,039	34,446,005	0.86
List	264	18	85	11	152,236,539	62,626,505	0.71
Mandelbrot	260	44	72	36	126,081,387	44,021,453	0.74
NBody	413	37	225	30	193,750,718	25,750,123	0.88
Permute	281	15	95	8	245,888,039	18,745,005	0.93
Queens	288	14	100	7	203,489,039	10,353,005	0.95
Sieve	250	17	68	10	494,367,039	67,233,005	0.88
Storage	260	13	74	6	135,163,039	6,827,005	0.95
Towers	301	17	116	10	177,028,239	12,304,205	0.94

Basic Operations

Basic operations are also known as primitives or built-in functions. We include comparisons, arithmetic and bit operations, reading the size of strings or arrays, and string operations. Since the complexity of these operations range from simple integer additions, which can be mapped directly to a processor instruction, up to trigonometric functions, or string comparisons, which require complex algorithms, we categorize them in groups with similar properties.

These groups are defined as follows:

Operands	Operations
ptr	=, !=
bool, int	<, >, =, <=, !=, >=
float	<, >, =, <=, !=, >=
bool, int	+, -, &, ^, !, <<, >>>
int	*, /, %, rem
float	+, -, *, /, round
float	sin, cos, sqrt
str	+, =, !=
str, arr	length
str	substring

The following plot gives an overview of the usage of these operations by the benchmarks.

The following table gives a numerical overview of the metrics.

	Group	Sites	Activations (per iter.)
CD	bool, int: <, >, =, <=, <>, >=	15	42,267,119
	ptr: ==, !=	11	4,598,236
	bool, int: +, -, &, <<, >>>, ^, !	14	3,648,020
	float: <, >, =, <=, <>, >=	54	38,482,634
	float: +, -, *, //, round	71	32,199,022
	float: sin, cos, sqrt	5	272,527
	str/arr: length	4	940,957
	int: *, /, %, rem	1	2,509
	str: +, =, !=	3	3
DeltaBlue	bool, int: <, >, =, <=, <>, >=	24	3,668,748
	ptr: ==, !=	16	2,640,155
	bool, int: +, -, &, <<, >>>, ^, !	32	3,764,511
	str/arr: length	8	360,490
	int: *, /, %, rem	5	300,185
	str: +, =, !=	3	3
Havlak	bool, int: <, >, =, <=, <>, >=	33	18,011,700
	ptr: ==, !=	6	3,174,541
	bool, int: +, -, &, <<, >>>, ^, !	45	22,316,780
	str/arr: length	15	6,094,993
	int: *, /, %, rem	4	2,072
	str: +, =, !=	3	3
Json	bool, int: <, >, =, <=, <>, >=	8	3,575,502
	ptr: ==, !=	5	500
	bool, int: +, -, &, <<, >>>, ^, !	17	6,849,503
	str/arr: length	7	3,117,900
	int: *, /, %, rem	2	60,300
	str: +, =, !=	45	13,536,203
	str: substring	2	2,848,100
Richards	bool, int: <, >, =, <=, <>, >=	11	10,001,402
	ptr: ==, !=	14	36,578,700
	bool, int: +, -, &, <<, >>>, ^, !	14	22,839,803
	str/arr: length	1	900
	int: *, /, %, rem	2	999,900
	str: +, =, !=	3	3
Bounce	bool, int: <, >, =, <=, <>, >=	6	37,806,002
	bool, int: +, -, &, <<, >>>, ^, !	16	29,431,503
	str/arr: length	1	76,500
	int: *, /, %, rem	5	1,200,000
	str: +, =, !=	3	3
List	bool, int: <, >, =, <=, <>, >=	3	54,002
	bool, int: +, -, &, <<, >>>, ^, !	5	61,503
	str: +, =, !=	3	3
Mandelbrot	bool, int: <, >, =, <=, <>, >=	8	9,065,196
	bool, int: +, -, &, <<, >>>, ^, !	14	9,447,050
	float: <, >, =, <=, <>, >=	1	8,195,047
	float: +, -, *, //, round	14	66,311,876
	str: +, =, !=	3	3
NBody	bool, int: <, >, =, <=, <>, >=	2	7,000,030
	bool, int: +, -, &, <<, >>>, ^, !	5	6,500,028
	float: <, >, =, <=, <>, >=	1	1
	float: +, -, *, //, round	69	77,500,206
	float: sin, cos, sqrt	2	2,500,010
	str/arr: length	5	1,750,007
	str: +, =, !=	3	3
Permute	bool, int: <, >, =, <=, <>, >=	4	17,328,002
	bool, int: +, -, &, <<, >>>, ^, !	7	22,365,003
	str/arr: length	1	1,000
	str: +, =, !=	3	3
Queens	bool, int: <, >, =, <=, <>, >=	2	11,472,002
	bool, int: +, -, &, <<, >>>, ^, !	10	43,111,003
	str/arr: length	1	40,000
	str: +, =, !=	3	3
Sieve	bool, int: <, >, =, <=, <>, >=	3	65,223,002
	bool, int: +, -, &, <<, >>>, ^, !	8	115,425,003
	str/arr: length	1	3,000
	str: +, =, !=	3	3
Storage	bool, int: <, >, =, <=, <>, >=	3	12,288,002
	bool, int: +, -, &, <<, >>>, ^, !	8	28,670,003
	str/arr: length	1	1,365,000
	int: *, /, %, rem	2	8,192,000
	str: +, =, !=	3	3
Towers	bool, int: <, >, =, <=, <>, >=	5	9,771,002
	bool, int: +, -, &, <<, >>>, ^, !	9	14,751,603
	str: +, =, !=	3	3