bench

Benchmark of Template Rendering

Run the benchmark (recycle):
https://raw.githack.com/nextapps-de/mikado/master/bench/

Run the benchmark (keyed):
https://raw.githack.com/nextapps-de/mikado/master/bench/#keyed

Run the benchmark (internal/data-driven):
https://raw.githack.com/nextapps-de/mikado/master/bench/#internal

There are 3 kinds of test scenarios:

1. recycle	In this mode all existing nodes could be reused (recycling nodes). This mode has no restrictions.
2. keyed	In this mode just existing nodes with the same key (ID) are allowed to be reused. Re-arrangement / reconcile is a rare implemented but also strong feature which is explicitly covered by the test "order".
3. data-driven	This mode runs through the same internal pool of data (same references, no new data from external or by creation) and compares the performance gain of data-driven libraries. Especially the test "update" and "repaint" covers the strength of this mode. This mode has no restrictions.

Whether a library provides optimizations to one of these modes or not, it is fair to compare each of them in a different scenario.

When the option "keep best run" is enabled it will replace the better result with the old one (separately for each test). When disabled, it will summarize the results for each test.

Keyed

Library	KB	RAM	Create	Replace	Update	Order	Repaint	Add	Remove	Toggle	Clear	Index	Score
mikado	3	22	19301	8535	206747	51470	220010	35346	27945	31265	26378	996	54089
mikado-proxy	8.3	30	10288	5901	27129	18648	28194	14912	19278	16526	26216	537	12803
solid	0	339	737	665	7291	4029	13279	1391	7487	2470	15227	149	3587
inferno	8.4	311	754	724	5493	5266	6055	1323	7443	2302	15982	191	2647
mithril	9.6	263	637	612	4599	4267	4997	1120	6614	2004	12622	170	2256
redom	2.9	522	421	411	4146	3719	4215	761	5750	1380	11744	190	1954
domc	4.5	393	1078	1059	1082	1129	1101	1128	2049	1464	24931	211	1250
innerhtml	0	494	1029	999	993	876	885	935	1769	1186	27131	154	1107
surplus	15.8	626	975	857	849	854	846	878	1560	1187	23713	162	987
sinuous	7.5	650	842	809	812	824	820	813	1577	1096	18047	159	941
jquery	31.3	684	809	707	703	643	652	698	1129	860	5520	84	708
lit-html	17.3	1179	441	411	409	413	409	431	761	550	4964	79	487
ractive	68.2	4684	165	156	158	158	159	166	298	212	1944	36	202
knockout	24.8	2657	91	67	68	68	68	84	130	103	1162	45	161

Non-Keyed (Recycle)

Library	KB	RAM	Create	Replace	Update	Order	Repaint	Add	Remove	Toggle	Clear	Index	Score
mikado	3	12	19344	8611	205552	52497	243874	34405	27858	31847	26483	995	18276
mikado-proxy	8.3	43	10041	5782	24215	18201	24873	14302	19126	16517	25778	494	5927
domc	4.5	119	1142	6524	51753	11723	94361	2395	22497	4617	25167	431	4510
lit-html	17.3	359	446	4505	28027	8252	44001	897	8317	1645	5021	197	2220
solid	0	267	741	2196	6302	3223	13560	1346	8397	2452	14964	162	1320
inferno	8.4	283	761	750	5469	5000	5988	1363	7366	2345	16151	185	1122
mithril	9.6	244	645	2134	4636	3126	5039	1179	7191	2011	12604	179	1034
redom	2.9	355	448	2416	4184	3397	4554	874	6620	1530	12348	214	1005
innerhtml	0	506	1059	1009	1018	890	883	901	1636	1186	27371	147	710
surplus	15.8	581	955	831	835	835	845	939	1540	1155	25324	162	671
ractive	68.2	1182	164	1514	5826	2619	8082	336	2970	616	1768	73	608
sinuous	7.5	641	838	802	803	817	810	814	1609	1075	16384	148	614
jquery	31.3	687	822	720	716	653	657	708	1136	872	5531	79	454
knockout	24.8	2576	91	67	69	68	69	85	130	103	1181	43	130

Data-Driven (Internal)

Library	KB	RAM	Create	Replace	Update	Order	Repaint	Add	Remove	Toggle	Clear	Index	Score
mikado-proxy	8.3	18	7350	2600	18467	11819	5601305	10254	23402	14499	18986	551	109601
mikado	3	15	18980	8374	78140	49034	260674	33723	27425	30768	25797	911	17362
sinuous	7.5	830	372	377	36533	38554	327320	757	13058	1529	12815	289	8431
domc	4.5	120	1045	4841	40118	8670	75694	2050	14914	3703	22609	373	4248
surplus	15.8	203	995	836	-failed-	22881	-failed-	1902	13561	3480	20871	311	2063
inferno	8.4	285	779	751	5303	4982	5901	1396	6100	2363	14419	184	1200
redom	2.9	1126	439	2425	4139	3386	4666	840	6389	1482	12790	214	1170
mithril	9.6	283	637	623	4517	4056	4858	1113	6664	1964	12248	168	1046
innerHTML	0	505	1049	999	1003	871	874	875	1675	1167	20810	131	794
jquery	31.3	548	798	692	690	642	646	677	1112	856	5553	84	530
lit-html	17.3	1117	439	406	409	408	415	439	800	554	5027	79	378
knockout	24.8	1815	89	64	-failed-	116	-failed-	129	320	191	1056	57	196
ractive	68.2	4758	148	145	145	138	139	150	243	187	1434	33	146

Test goal

This stress test focuses a real life use case, where new data is coming from a source to be rendered through a template (e.g. from a server or by creation during runtime). The different to other benchmark implementations is, that the given task is not known before the data was available.

This test measures the raw rendering performance. If you look for a benchmark which covers more aspects goto here:
https://krausest.github.io/js-framework-benchmark/current.html

Angular and React?

There are some reasons why you did not find Angular or React on this list. Both aren't a library nor a framework, they are full integrated ecosystems which is quite unfair to compare. Also they runs completely async. If there was a way to force running them in sync I'm pretty sure, they would fill the bottom lines of this benchmark test.

Local Installation

Go to the folder bench/ and install dependencies:

npm install

Start local webserver:

npm run server

Go to the URL which displays in the console, e.g. http://localhost. The tests will start automatically. Results may differ through various browsers and OS.

Score

The score is calculated in relation to the median value of each test. That will keeping the benchmark factor between each library effectively but also vary relationally when new libraries were added.

Score = Sumtest(lib_ops / median_ops) / test_count * 1000

The file size and memory gets less relevance by applying the square root of these values.

Index

The score index is a very stable representation where each score points to a specific place in a ranking table. The maximum possible score and also the best place is 1000, that requires a library to be best in each category (regardless of how much better the factor is, that's the difference to the score value).

Index = Sumtest(lib_ops / max_ops) / test_count * 1000

The file size and memory gets less relevance by applying the square root of these values.

Tests

Test	Description
KB	The size of the library. This value has less relevance to the score calculation (uses square root).
RAM	The amount of memory which was allocated through the whole test (memory per cycle). This value also has less relevance to the score calculation (uses square root).
Create	The creation of 100 elements from blank.
Replace	The replacement of 100 new elements over 100 existing elements.
Update	Update content fields (order of items stay unchanged).
Arrange	Test re-arrangement, all contents stay unchanged. Toggles between: 1. swap second and fore-last item 2. re-arrange (4 shifted groups) 3. shuffle (on every 5th loop)
Repaint	Render same items in the same order.
Remove	The remove of 50 items from a list of 100.
Append	Append 50 new items to a list of 50 existing.
Toggle	Toggles between "Remove" and "Append" (test for optimizations like: pagination, content folding, list resizing).
Clear	Remove all items from a list of 100 existing.

How this benchmark work

Basically each framework provide one public method, which handles and render the incoming data, e.g.:

suite["domc"] = function(items){
    scope.items = items;
    root.update(scope);
};

or

suite["sinuous"] = function(items){
    data(items);
};

The data is unknown, the library does not know if data was added, removed, updated or stay unchanged before it gets the data. That's the main different to other benchmark implementations, where a programmer can iteratively solve a problem to a known task.

Regardless the function is doing, every test has to run through the same logic.

Random item factory

The items were created by a random factory. The items comes from a pre-filled pool (5 slots a 100 items), so that keyed libraries get a chance to match same IDs.

Also the items has some fields, which aren't included by the template. That is also important, because in this situation is the most common. Most other benchmarks just provide data which is consumed by the template.

Mimic data from a server or created during runtime

The items will be cloned before every test to mimic a fresh fetch from the server or the creation of new items during runtime. The "data-driven" mode disables cloning and perform changes over and over through the same data.

Dedicated sandbox

Each test suite runs in its own dedicated sandbox through an iframe. This is reducing noise from externals to a minimum.

Hidden render target

You may see benchmarks which draws the rendering visible to the users screen. It depends on what is the desired part to benchmark. This benchmark will just cover the raw time of applied re-calculations (creation/updating/removing of elements), the time the browser takes to make them visible is:

1. no part of this test
2. not relevant, because this time is almost not influenced by the library
3. introduce unnecessary distortion to the test

About requirements for tested libraries

1. Each library should provide at least its own features to change DOM. A test implementation should not force to implement something like node.nodeValue = "..." or node.className = "..." by hand. The goal is to benchmark library performance and not the performance made by an implementation of a developer. That is probably the biggest different to other benchmark tests.

2. Also asynchronous/scheduled rendering is not allowed.

3. The keyed test requires a largely non-reusing paradigm. When a new item comes from the outside, the library does not reusing nodes (on different keys/IDs).

About the test environment

This test also covers runtime optimizations of each library which is very important to produce meaningful results.

About median values

Using the median value is very common to normalize the spread in results in a statistically manner. But using the median as benchmark samples, especially when code runs through a VM, the risk is high that the median value is getting back a false result. One thing that is often overseen is the run of the garbage collector, which has a significantly cost and runs randomly. A benchmark which is based on median results will effectively cut out the garbage collector and may produce wrong results. A benchmark based on a best run will absolutely cut off the garbage collector.

This test implementation just using a median to map the results into a normalized scoring index. The results are based on the full computation time including the full run of the garbage collector. That also comes closest to a real environment.

About benchmark precision

It is not possible to provide absolute stable browser measuring. There are so many factors which has an impact of benchmarking that it makes no sense in trying to make "synthetic" fixes on things they cannot been fixed. Also every synthetic change may lead into wrong results and false interpreting. For my personal view the best benchmark just uses the browser without any cosmetics. That comes closest to the environment of an user who is using an application.