4.1.2 One to One Test
This test was made to compare the Arrowhead Framework with current communication frameworks in distributed systems, on the simplest of application conditions, on the simplest of metrics.
The two other solutions we chose for this were MQTT and DDS. The representatives for these technologies for this test are:
- For MQTT: the Eclipse Mosquitto server with clients written using Eclipse Paho
- For DDS: RTI DDS Connext
- And for Arrowhead: The server reference implementation and client frameworks
The test setup is incredibly simple:
One client wants a file, the other one has it. They use their respective frameworks to make the transfer. For convenience, the client which sends the file is always the Sender, and the one that receives it is always the Receiver, regardless of what specific role or roles they have in their respective frameworks' terminologies.
The Sender sends the file, and when the Receiver receives it, it sends back some kind of acknowledgement. The Sender measures the time between starting the sending process and receiving the acknowledgement. From this, and knowing the size of the file, we can calculate average round-trip times and bandwidths.
Firstly we pseudo randomly decide to which pair clients will do this test cycle. This is done to equalize the effects of memory cashing between the tested. The Sender, right after receiving the order to start the test makes a timestamp (number 0) and stores it. Then, sends the file, then receives acknowledgement, then waits two seconds, then makes another timestamp and stores it. This is repeated 12 times, making 13 timestamps. The waiting is done to let the network (and network controller) to clear up, and done at that time so that it's between every pair of timestamp. We consider, for test number x, that the start of the transfer was at timestamp number x-1, and the end at timestamp number x.
For each transfer, we first calculate the round-trip time from the its corresponding start and end timestamps, subtracting the two second wait from the duration. Knowing the size of the file sent, we calculate bandwidth by dividing the file size with the round-trip time. This is fairly bogus, but is useful for comparisons. Then, the five values; file size (bytes), start time (ISO-8601), end time (ISO-8601), round-trip-time (milliseconds), and bandwidth (bytes/second), in a .csv file with a name corresponding to the framework tested.
Then we repeat the whole process, until we have enough data.
The reason to sending the file 12 times each test, is because the Arrowhead Framework discovers the other party, while the others don't, meaning the Arrowhead client only has to touch the server once, while the other two has to use their framework for each transaction. For the Arrowhead client, this means that its numbers are worse on the first of the 12, but better on each subsequent one, meaning the higher this repetition number is, the better it looks for the Arrowhead Framework. But how many times would a real system, on average, use the same service? Sadly we found no publications relating to this, so I decided upon 12, for it's a highly composite number.
The raw data can be found in Tests/OneToOneTest/Out.
Just the simplest case for base numbers.
This test was ran with the following hardware, on the same wi-fi network.
- The servers ran on a i7-8565U @ 1.80 GHz and an Intel wireless-AC 9260 160 MHz
- The receivers ran on a i5-6200U @ 2.3 GHz and an Intel wireless-AC 3160
- The senders ran on a i5-7200U @ 2.5 GHz and an Intel wireless-AC 8265
From a complexity standpoint, MQTT is by far the easiest to work with, and that is on purpose. RTI DDS is very granular, but that is to provide QoS options. Making a client to the Arrowhead Framework is a lot of work, and while that's partially because it's very much in development, if all the features are done being developed, it will be the most powerful one.
On this test we had two goals: To see frameworks over public internet, and to find out if there's a breakpoint in performance around the 64 KB mark, as that is the maximum size of a TCP segment.
This test was run on the following hardware:
- The server ran on an Amazon Web Service
- The senders ran on a i7-8565U @ 1.80 GHz and an Intel wireless-AC 9260 160 MHz
- The receivers ran on a i5-6200U @ 2.3 GHz and an Intel wireless-AC 3160
The speed of the DDS connection comes down to the fact that it never needs to go to Germany, conversely the speed of the MQTT messages is so slow because they need to do it every time. The Arrowhead line is right in between them, because it needs to do so only 1/12th of the time.
On the other hand there doesn't seem to be a breakpoint at 64 KB.