diff --git a/src/program/lwaftr/doc/README.counters.md b/src/program/lwaftr/doc/README.counters.md new file mode 100644 index 0000000000..0a41fa4fc5 --- /dev/null +++ b/src/program/lwaftr/doc/README.counters.md @@ -0,0 +1,112 @@ +# Counters + +In order to better understand the flow of packets through the lwAftr app at +runtime, a number of counters are embedded in the code. They record the +number of packets and bytes handled at various points of the execution flow, +shown in the diagrams below. + +The counters' values can be accessed by means of the `snabb top` subcommand. + +## Execution flow + +Here is the lwAftr's overall execution flow: + +![main flow](images/main-flow.png) + +Packets coming from the b4 on users' premises are decapsulated, handled, then +sent to the Internet or dropped, as appropriate. On the other side, packets +coming from the Internet are handled, possibly dropped, or encapsulated and +sent to users' b4. + +Some packets coming from a b4 may be destined to another b4 handled by the same +lwAftr instance: in that case, as an optimization, they are short-circuited +("hairpinned") to their destination internally, so that they are not uselessly +routed forward and back. + +Each direction is broken in two by lookup queues, in order to reduce the cost +of lookups in the binding table. The four resulting macro blocks are detailed +below, in clockwise order. + +For each macro block, the place of all counters in the execution flow is first +shown graphically, then each counter is described in detail. Several counters +appear in more than one place, and the dashed blocks designate functions in +the Lua code. + +### b4 to decapsulation queue + +![b4 to decapsulation queue](images/b4-to-decaps-queue.png) + +Counters: + +- **drop-misplaced-not-ipv6**: non-IPv6 packets incoming on the IPv6 link +- **in-ipv6**: all valid incoming IPv6 packets +- **drop-unknown-protocol-ipv6**: packets with an unknown IPv6 protocol +- **drop-in-by-policy-icmpv6**: incoming ICMPv6 packets dropped because of + current policy +- **out-icmpv4**: internally generated ICMPv4 error packets +- **out-ipv4**: all valid outgoing IPv4 packets +- **drop-too-big-type-but-not-code-icmpv6**: the packet's ICMP type was + "Packet too big", but its ICMP code was not an acceptable one for this type +- **drop-over-time-but-not-hop-limit-icmpv6**: the packet's time limit was + exceeded, but the hop limit was not +- **drop-unknown-protocol-icmpv6**: packets with an unknown ICMPv6 protocol + +### decapsulation queue to Internet + +![decapsulation queue to internet](images/decaps-queue-to-internet.png) + +Counters: + +- **drop-no-source-softwire-ipv6**: no matching source softwire in the binding + table; incremented whether or not the reason was RFC7596 +- **out-ipv4**: all valid outgoing IPv4 packets +- **hairpin-ipv4**: IPv4 packets going to a known b4 (hairpinned) +- **drop-out-by-policy-icmpv6**: internally generated ICMPv6 error packets + dropped because of current policy +- **drop-over-rate-limit-icmpv6**: packets dropped because the outgoing ICMPv6 + rate limit was reached +- **out-icmpv6**: internally generated ICMPv6 error packets + +### Internet to encapsulation queue + +![internet to encapsulation queue](images/internet-to-encaps-queue.png) + +Counters: + +- **drop-in-by-policy-icmpv4**: incoming ICMPv4 packets dropped because of + current policy +- **in-ipv4**: all valid incoming IPv4 packets +- **drop-misplaced-not-ipv4**: non-IPv4 packets incoming on the IPv4 link +- **drop-bad-checksum-icmpv4**: ICMPv4 packets dropped because of a bad + checksum +- **drop-all-ipv4-iface**, **drop-all-ipv6-iface**: all dropped packets and + bytes that came in over the IPv4/6 interfaces, whether or not they're + actually IPv4/6 (they only include data about packets that go in/out over the + wires, excluding internally generated ICMP packets) + +### Encapsulation queue to b4 + +![encapsulation queue to b4](images/encaps-queue-to-b4.png) + +Counters: + +- **out-ipv6**: all valid outgoing IPv6 packets +- **drop-over-mtu-but-dont-fragment-ipv4**: IPv4 packets whose size exceeded + the MTU, but the DF (Don't Fragment) flag was set +- **drop-ttl-zero-ipv4**: IPv4 packets dropped because their TTL was zero +- **drop-out-by-policy-icmpv4**: internally generated ICMPv4 error packets + dropped because of current policy +- **drop-no-dest-softwire-ipv4**: no matching destination softwire in the + binding table; incremented whether or not the reason was RFC7596 +- **drop-in-by-rfc7596-icmpv4**: incoming ICMPv4 packets with no destination + (RFC 7596 section 8.1) + +## Notes + +The internally generated ICMPv4 error packets that are then dropped because +of policy are not recorded as dropped: only incoming ICMP packets are. + +Implementation detail: rhe counters can be accessed as files in the runtime +area of the Snabb process, typically under +`/var/run/snabb/[PID]/app/lwaftr/counters/`. Most of them are represented by +two files, ending with the `bytes` and `packets` suffixes. diff --git a/src/program/lwaftr/doc/README.md b/src/program/lwaftr/doc/README.md index a56b5e68cf..27eb608753 100644 --- a/src/program/lwaftr/doc/README.md +++ b/src/program/lwaftr/doc/README.md @@ -112,6 +112,8 @@ get the lwAFTR working on virtualized network interfaces. [Troubleshooting](./README.troubleshooting.md) +[Counters](./README.counters.md) + ## Performance [Benchmarking](./README.benchmarking.md) @@ -125,4 +127,3 @@ get the lwAFTR working on virtualized network interfaces. [Discovery of next-hop L2 addresses via NDP](./README.ndp.md) [Change Log](./CHANGELOG.md) - diff --git a/src/program/lwaftr/doc/genbook.sh b/src/program/lwaftr/doc/genbook.sh index 617aff7659..421fa9cdf0 100755 --- a/src/program/lwaftr/doc/genbook.sh +++ b/src/program/lwaftr/doc/genbook.sh @@ -39,6 +39,8 @@ $(cat README.rfccompliance.md) $(cat README.troubleshooting.md) +$(cat README.counters.md) + $(cat README.breaking_changes.md) EOF diff --git a/src/program/lwaftr/doc/images/b4-to-decaps-queue.dia b/src/program/lwaftr/doc/images/b4-to-decaps-queue.dia new file mode 100644 index 0000000000..c494c35af0 Binary files /dev/null and b/src/program/lwaftr/doc/images/b4-to-decaps-queue.dia differ diff --git a/src/program/lwaftr/doc/images/b4-to-decaps-queue.png b/src/program/lwaftr/doc/images/b4-to-decaps-queue.png new file mode 100644 index 0000000000..f34e121625 Binary files /dev/null and b/src/program/lwaftr/doc/images/b4-to-decaps-queue.png differ diff --git a/src/program/lwaftr/doc/images/decaps-queue-to-internet.dia b/src/program/lwaftr/doc/images/decaps-queue-to-internet.dia new file mode 100644 index 0000000000..a2b67b699a Binary files /dev/null and b/src/program/lwaftr/doc/images/decaps-queue-to-internet.dia differ diff --git a/src/program/lwaftr/doc/images/decaps-queue-to-internet.png b/src/program/lwaftr/doc/images/decaps-queue-to-internet.png new file mode 100644 index 0000000000..eb25c6314c Binary files /dev/null and b/src/program/lwaftr/doc/images/decaps-queue-to-internet.png differ diff --git a/src/program/lwaftr/doc/images/encaps-queue-to-b4.dia b/src/program/lwaftr/doc/images/encaps-queue-to-b4.dia new file mode 100644 index 0000000000..3d6f7dfdb7 Binary files /dev/null and b/src/program/lwaftr/doc/images/encaps-queue-to-b4.dia differ diff --git a/src/program/lwaftr/doc/images/encaps-queue-to-b4.png b/src/program/lwaftr/doc/images/encaps-queue-to-b4.png new file mode 100644 index 0000000000..af03d51267 Binary files /dev/null and b/src/program/lwaftr/doc/images/encaps-queue-to-b4.png differ diff --git a/src/program/lwaftr/doc/images/internet-to-encaps-queue.dia b/src/program/lwaftr/doc/images/internet-to-encaps-queue.dia new file mode 100644 index 0000000000..2d147b558c Binary files /dev/null and b/src/program/lwaftr/doc/images/internet-to-encaps-queue.dia differ diff --git a/src/program/lwaftr/doc/images/internet-to-encaps-queue.png b/src/program/lwaftr/doc/images/internet-to-encaps-queue.png new file mode 100644 index 0000000000..92d49be2c8 Binary files /dev/null and b/src/program/lwaftr/doc/images/internet-to-encaps-queue.png differ diff --git a/src/program/lwaftr/doc/images/main-flow.dia b/src/program/lwaftr/doc/images/main-flow.dia new file mode 100644 index 0000000000..6338cf573a Binary files /dev/null and b/src/program/lwaftr/doc/images/main-flow.dia differ diff --git a/src/program/lwaftr/doc/images/main-flow.png b/src/program/lwaftr/doc/images/main-flow.png new file mode 100644 index 0000000000..eec84d7564 Binary files /dev/null and b/src/program/lwaftr/doc/images/main-flow.png differ