Scalability Benchmark Write Up

Device Performance Ranking

Tests were done with 1500 fake devices, each with 18 points. Only the "depth first all" for each device was published. Instructions for how to setup and run this benchmark are in scripts/scalability-testing/README.md.

This test is meant to push devices way beyond what is normally expected of the platform to measure how well each device can be expected to scale. Poor performance on this test does not mean that the device is unsuited for use with the platform. For instance, all listed devices are more than adequate for retrieving data from 20 real devices with 20 data point and forwarding the results to another platform.

Time is measured from the start of the first device publish to the completion of the last device publish. Data was published once per minute. DNF means that the device could not complete the task before the next publish started.

All values are in seconds.

Device	Storage	Publish Only	Publish One listener	Historian V2 Catch Up
Xeon CPU W3503 (2.4GHz 2 cores) aka "Franklin"	Hard Drive	1.76	3.5	~6
6 Core VM on Xeon CPU E3-2603 (1.8GHZ 8 cores)	SSD Drive	5.8	9.3	~5
Odroid XU4	eMMC	5.8	8.02	~5
Intel Compute Stick	eMMC	8.6	12.78	~7.5
Minnowboard Max	SD UHS-3	16.14	28.7	~14
Banana Pi	SD UHS-3	17.76	36.16	~25
RaspberryPi 2 Model B	SD UHS-3	18.03	22.29	~25
SMC TWT controller	SD	36.69	DNF	DNF
BeagleBone2	SD	47.99	DNF	DNF
BeagleBoneBlack	SD UHS-3	49.87	DNF	DNF

Takeaways

Drive speed has a definite affect on the base historian's ability to keep up with load. Even though Franklin dominated performance tests the historian was slightly slower than the Odroid and VM which both use a form of high speed flash storage instead of mechanical storage.

Overhead from running in a VM is significant.

Other Observations

All tests were run at the DEBUG logging level. On some of the higher speed devices redirecting this output to a file can improve performance. For instance, the completion time on Franklin for "Publish One Listener" drops from 3.5 seconds to 2.9 when console output is redirected to a file. On slower devices it did not appear to make much difference.

The length of the publish time hinges on the number of publishes and not on the size of each message. Reducing the number of points on each device has little effect on the total publish time.

When the number of fake devices is reduced to 500 with 6 points each, all devices are able to keep up.

Device	Storage	Publish Only	Publish One listener	Historian V2 Catch Up
Minnowboard Max	SD UHS-3	5.24	11.83	<1
Banana Pi	SD UHS-3	5.82	13.02	<1
RaspberryPi 2 Model B	SD UHS-3	6.00	12.10	~3
SMC TWT controller	SD	11.61	22.96	~1
BeagleBone2	SD	15.7	26.42	~10
BeagleBoneBlack	SD UHS-3	16.94	26.41	~11