Merge pull request snabbco#34 from Igalia/lwaftr-docs

Added documentation on the lwaftr

README.lwaftr.md

@@ -0,0 +1 @@
+For information about the lwaftr, please see the docs/ directory.

docs/README.benchmarking.md

@@ -0,0 +1,217 @@
+# Benchmarking
+
+The instructions in [README.first.md](README.first.md) for running the lwaftr with a load 
+generator are the instructions for the primary way to benchmark the lwaftr. 
+In short:
+
+To run two load generators and one lwaftr, you will need four interfaces. The 
+following example assumes that `01:00.0` is cabled to `01:00.1`, and that 
+`02:00.0` is cabled to `02:00.1`. Change the concrete pci devices specified to 
+match the current system; See [README.first.md](README.first.md).
+
+```bash
+$ cd ${SNABB_LW_DIR} # The directory snabb-lwaftr is checked out into
+$ sudo ./bin/snabb-lwaftr-blaster \
+    --v4-pcap tests/apps/lwaftr/benchdata/ipv4-0550.pcap \
+    --v6-pcap tests/apps/lwaftr/benchdata/ipv6-0550.pcap \
+    --v4-pci 01:00.0 --v6-pci 02:00.0
+```
+
+Now, run the lwaftr itself:
+```bash
+$ sudo ./bin/snabb-lwaftr \
+    --bt tests/apps/lwaftr/data/binding.table \
+    --conf tests/apps/lwaftr/data/icmp_on_fail.conf \
+    --v4-pci 01:00.1 --v6-pci 02:00.1
+```
+
+By varying the `--v4-pcap` and `--v6-pcap` arguments, the performance of the 
+lwaftr can be benchmarked with different types of loads.
+
+The contents of the pcap file are fed repeatedly through each NIC, to the 
+lwaftr, by the load generator.
+
+## Current performance
+
+```bash
+$ for x in {1..10}; do 
+   sudo ./snabb snsh ./apps/lwaftr/nic_ui.lua \
+      -v -D 5 \
+      ../tests/apps/lwaftr/data/binding.table \
+      ../tests/apps/lwaftr/data/icmp_on_fail.conf \
+      0000:01:00.1 0000:02:00.1
+  ; done | ../tests/apps/lwaftr/end-to-end/lwstats.py 
+Initial v4 MPPS: min: 1.443, max: 2.153, avg: 1.875, stdev: 0.3520 (n=10)
+Initial v4 Gbps: min: 5.887, max: 8.786, avg: 7.649, stdev: 1.4364 (n=10)
+Initial v6 MPPS: min: 1.443, max: 2.014, avg: 1.791, stdev: 0.2802 (n=10)
+Initial v6 Gbps: min: 6.810, max: 9.506, avg: 8.454, stdev: 1.3227 (n=10)
+Final v4 MPPS: min: 1.468, max: 2.178, avg: 1.903, stdev: 0.3554 (n=10)
+Final v4 Gbps: min: 5.991, max: 8.885, avg: 7.762, stdev: 1.4498 (n=10)
+Final v6 MPPS: min: 1.468, max: 2.036, avg: 1.818, stdev: 0.2821 (n=10)
+Final v6 Gbps: min: 6.930, max: 9.609, avg: 8.578, stdev: 1.3315 (n=10)
+```
+
+This does 10 runs of 5 seconds each, and reports on the _minimum_/_maximum_/
+_average_ performance of the first second (_Initial_) and last second (_Final_).
+Performance tends to increase with time, although this effect disappears with 
+the highest speeds.
+
+The MPPS and Gbps speeds are relative to the outgoing interface, which is why 
+the same MPPS figures correspond to different Gbps. Both interfaces are being 
+fed with 550 byte packets, but the outgoing v6 ones are 590 bytes (due to 
+encapsulation), while the outgoing v4 ones are 510 bytes (they have undergone 
+decapsulation).
+
+Maximum speeds on all interfaces during both initial and final runs were over 
+2 MPPS.
+
+## Understanding current performance
+
+Performance varies within and between runs. It tends to increase with time,
+although this effect disappears with the highest speeds.
+
+### Performance of a slow run
+
+```bash
+$ sudo ./snabb snsh ./apps/lwaftr/nic_ui.lua \
+    -v -D 25 \
+    ../tests/apps/lwaftr/data/binding.table \
+    ../tests/ps/lwaftr/data/icmp_on_fail.conf \
+    0000:01:00.1 0000:02:00.1
+v4_stats: 1.476 MPPS, 6.021 Gbps.
+v6_stats: 1.476 MPPS, 6.965 Gbps.
+v4_stats: 1.500 MPPS, 6.121 Gbps.
+v6_stats: 1.500 MPPS, 7.082 Gbps.
+v4_stats: 1.499 MPPS, 6.118 Gbps.
+v6_stats: 1.499 MPPS, 7.077 Gbps.
+v4_stats: 1.503 MPPS, 6.130 Gbps.
+v6_stats: 1.503 MPPS, 7.092 Gbps.
+v4_stats: 1.503 MPPS, 6.131 Gbps.
+v6_stats: 1.503 MPPS, 7.093 Gbps.
+v4_stats: 1.503 MPPS, 6.130 Gbps.
+v6_stats: 1.503 MPPS, 7.092 Gbps.
+v4_stats: 1.503 MPPS, 6.130 Gbps.
+v6_stats: 1.503 MPPS, 7.092 Gbps.
+v4_stats: 1.502 MPPS, 6.127 Gbps.
+v6_stats: 1.502 MPPS, 7.088 Gbps.
+v4_stats: 1.501 MPPS, 6.123 Gbps.
+v6_stats: 1.501 MPPS, 7.083 Gbps.
+v4_stats: 1.498 MPPS, 6.112 Gbps.
+v6_stats: 1.498 MPPS, 7.071 Gbps.
+v4_stats: 1.499 MPPS, 6.115 Gbps.
+v6_stats: 1.499 MPPS, 7.074 Gbps.
+v4_stats: 1.502 MPPS, 6.128 Gbps.
+v6_stats: 1.502 MPPS, 7.089 Gbps.
+v4_stats: 1.502 MPPS, 6.128 Gbps.
+v6_stats: 1.502 MPPS, 7.090 Gbps.
+```
+
+This needs further investigation. Note that the speed trends upward with time.
+
+### Performance of a mixed run
+
+```bash
+$ sudo ./snabb snsh ./apps/lwaftr/nic_ui.lua \
+    -v -D 25 \
+    ../tests/apps/lwaftr/data/binding.table \
+    ../tests/apps/lwaftr/data/icmp_on_fail.conf \
+    0000:01:00.1 0000:02:00.1
+v4_stats: 2.142 MPPS, 8.740 Gbps.
+v6_stats: 2.003 MPPS, 9.456 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.884 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.177 MPPS, 8.884 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 1.720 MPPS, 7.019 Gbps.
+v6_stats: 1.656 MPPS, 7.815 Gbps.
+v4_stats: 1.635 MPPS, 6.669 Gbps.
+v6_stats: 1.585 MPPS, 7.479 Gbps.
+v4_stats: 1.339 MPPS, 5.462 Gbps.
+v6_stats: 1.339 MPPS, 6.318 Gbps.
+v4_stats: 1.338 MPPS, 5.460 Gbps.
+v6_stats: 1.338 MPPS, 6.317 Gbps.
+v4_stats: 2.177 MPPS, 8.883 Gbps.
+v6_stats: 2.036 MPPS, 9.610 Gbps.
+```
+
+The speed dips and recovers; this has not yet been investigated.
+
+### Performance of a steady fast run
+
+```bash
+$ sudo ./snabb snsh ./apps/lwaftr/nic_ui.lua \
+    -v -D 25 \
+    ../tests/apps/lwaftr/data/binding.table \
+    ../tests/ps/lwaftr/data/icmp_on_fail.conf \
+    0000:01:00.1 0000:02:00.1
+v4_stats: 2.150 MPPS, 8.773 Gbps.
+v6_stats: 2.011 MPPS, 9.492 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.884 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.177 MPPS, 8.884 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.177 MPPS, 8.884 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.884 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+v4_stats: 2.178 MPPS, 8.885 Gbps.
+v6_stats: 2.036 MPPS, 9.609 Gbps.
+```
+
+This is a fast, solid run, with extremely consistent speeds.
+
+## Approximate benchmarking, without physical NICs
+
+To get an idea of the raw speed of the lwaftr without interaction with NICs, 
+or check the impact of changes on a development machine that may not have 
+Intel 82599 NICs, `benchui.lua` may be used:
+
+```bash
+$ sudo ./snabb snsh ./apps/lwaftr/benchui.lua \
+    ../tests/apps/lwaftr/data/binding.table \
+    ../tests/apps/lwaftr/data/icmp_on_fail.conf \
+    ../tests/apps/lwaftr/benchdata/ipv4-0550.pcap \
+    ../tests/apps/lwaftr/benchdata/ipv6-0550.pcap 
+statisticsv6: 4.101 MPPS, 19.358 Gbps.
+statisticsv4: 4.101 MPPS, 16.734 Gbps.
+statisticsv6: 4.128 MPPS, 19.486 Gbps.
+statisticsv4: 4.128 MPPS, 16.844 Gbps.
+statisticsv6: 4.126 MPPS, 19.474 Gbps.
+statisticsv4: 4.126 MPPS, 16.833 Gbps.
+statisticsv6: 4.127 MPPS, 19.481 Gbps.
+statisticsv4: 4.127 MPPS, 16.840 Gbps.
+```
+
+The processing is not limited to 10 Gbps, as no NIC hardware is involved.

docs/README.bindingtable.md

@@ -0,0 +1,40 @@
+# The Binding Table
+
+Snabb-lwaftr alpha specifies binding tables in text files.
+
+```bash
+$ cat binding.table
+{
+ {'127:2:3:4:5:6:7:128', '178.79.150.233', 1, 100, '8:9:a:b:c:d:e:f'},
+ {'127:11:12:13:14:15:16:128', '178.79.150.233', 101, 64000},
+ {'127:22:33:44:55:66:77:128', '178.79.150.15', 5, 7000},
+ {'127:24:35:46:57:68:79:128', '178.79.150.2', 7800, 7900, '1E:1:1:1:1:1:1:af'},
+ {'127:14:25:36:47:58:69:128', '178.79.150.3', 4000, 5050, '1E:2:2:2:2:2:2:af'}
+}
+```
+
+The format is a [Lua table](http://www.lua.org/pil/2.5.html), containing more Lua tables.
+Each of these subtables (each shown on one line, above) has several parts. As an 
+example, look at the following entry:
+
+```lua
+{'127:24:35:46:57:68:79:128', '178.79.150.2', 7800, 7900, '1E:1:1:1:1:1:1:af'}
+```
+
+Where:
+
+* **127:24:35:46:57:68:79:128** is the IPv6 address of a B4.
+* **178.79.150.2** is the IPv4 address of the same B4 (not necessarily unique).
+* **7800** and **7900** are the start and end of the port range on that IPv4
+address that are assigned to that B4.
+* **1E:1:1:1:1:1:1:af** is the IPv6 address associated with the lwaftr for this
+binding table entry. It is optional, and if it is not specified, the default
+configured lwaftr IPv6 address is used.
+
+In this example binding table, the default is used for the first three entries,
+and a custom address is specified for the last two entries.
+
+Entries must be comma-separated. Having a comma after the last entry is optional.
+
+The table is a Lua data structure, not a line-oriented format, but keeping
+one entry per line aids human readability.

docs/README.build.md

@@ -0,0 +1,24 @@
+# How to build snabb-lwaftr
+
+## Step 1: Fetch the sources
+
+```bash
+$ git clone https://github.com/Igalia/snabbswitch.git
+```
+
+## Step 2: Check out the lwaftr development branch:
+
+```bash
+$ cd snabbswitch && git checkout lwaftr_nutmeg
+```
+
+## Step 3: Build
+
+```bash
+$ make
+```
+
+Note that this requires internet access to fetch the submodules.
+
+This is all that is needed to build snabb-lwaftr. See `bin/` for **snabb-lwaftr**,
+and see [README.first.md](README.first.md) for instructions on how to use it.

docs/README.configuration.md

@@ -0,0 +1,98 @@
+# Snabb-lwaftr configuration
+
+Snabb-lwaftr alpha is configured by a text file.
+
+```bash
+$ cat sample.conf
+aftr_ipv4_ip = ipv4:pton("10.10.10.10"),
+aftr_ipv6_ip = ipv6:pton('8:9:a:b:c:d:e:f'),
+aftr_mac_b4_side = ethernet:pton("22:22:22:22:22:22"),
+aftr_mac_inet_side = ethernet:pton("12:12:12:12:12:12"),
+b4_mac = ethernet:pton("44:44:44:44:44:44"),
+binding_table = bt.get_binding_table(),
+hairpinning = true,
+icmpv6_rate_limiter_n_packets=3e5,
+icmpv6_rate_limiter_n_seconds=4,
+inet_mac = ethernet:pton("68:68:68:68:68:68"),
+ipv4_mtu = 1460,
+ipv6_mtu = 1500,
+policy_icmpv4_incoming = policies['ALLOW'],
+policy_icmpv6_incoming = policies['ALLOW'],
+policy_icmpv4_outgoing = policies['ALLOW'],
+policy_icmpv6_outgoing = policies['ALLOW']
+```
+
+The lwaftr is associated with two physical network cards. One of these cards
+faces the internet; traffic over it is IPv4. The other faces the IPv6-only
+internal network, and communicates primarily with B4s.
+
+## Line-by-line explanation
+
+First, the IP and MAC addresses for both interfaces are set:
+
+```lua
+aftr_ipv4_ip = ipv4:pton("10.10.10.10"),
+aftr_ipv6_ip = ipv6:pton('8:9:a:b:c:d:e:f'),
+aftr_mac_b4_side = ethernet:pton("22:22:22:22:22:22"),
+aftr_mac_inet_side = ethernet:pton("12:12:12:12:12:12"),
+```
+
+This associates **12:12:12:12:12:12** and **10.10.10.10** with the
+internet-facing NIC, and **8:9:a:b:c:d:e:f** and **22:22:22:22:22:22** with the
+NIC facing the internal network.
+
+As this software is alpha and built on a kernel bypass basis, it does not have
+support for [ARP](https://en.wikipedia.org/wiki/Address_Resolution_Protocol) or
+[NDP](https://en.wikipedia.org/wiki/Neighbor_Discovery_Protocol). It assumes that
+it will talk directly to only one host on each side, and specifies their MAC
+addresses for the outgoing packets.
+
+```lua
+b4_mac = ethernet:pton("44:44:44:44:44:44"),
+inet_mac = ethernet:pton("68:68:68:68:68:68"),
+```
+
+The alpha lwaftr can talk to any host, but assumes that the above ones are the
+next hop.
+
+```lua
+binding_table = bt.get_binding_table(),
+```
+
+See [README.bindingtable.md](README.bindingtable.md) for binding table details.
+
+```lua
+hairpinning = true,
+```
+
+Configurable hairpinning is a requirement of [RFC 7596](https://tools.ietf.org/html/rfc7596);
+it can be true or false.
+
+```lua
+icmpv6_rate_limiter_n_packets=3e5,
+icmpv6_rate_limiter_n_seconds=4,
+```
+
+ICMPv6 rate limiting is mandated by several RFCs. This example says that the
+lwaftr can send at most 300,000 (3 * 10^5) ICMPv6 packets per 4 seconds.
+Lower values are recommended for non-experimental use.
+
+```lua
+ipv4_mtu = 1460,
+ipv6_mtu = 1500,
+```
+
+The MTU settings are used to determine whether a packet needs to be fragmented.
+The current MTU handling is otherwise underdeveloped. It is not dynamically
+updated on receiving ICMP packet too big messages.
+
+```lua
+policy_icmpv4_incoming = policies['ALLOW'],
+policy_icmpv6_incoming = policies['ALLOW'],
+policy_icmpv4_outgoing = policies['ALLOW'],
+policy_icmpv6_outgoing = policies['ALLOW']
+```
+
+Snabb-lwaftr can be configured to ALLOW or DROP incoming and outgoing ICMPv4
+and ICMPv6 messages. If a finer granularity of control is desired, contact the
+development team via github or email.