dispatcher removal

change last-mile router port forwarding

removing dispatcher from snet and infra libraries

intermediate commit remove dispatcher

fix dataplane port parsing

adapt end2end

braccept UTs

integration no probe

add port range

fix port range

remove ref to dispatcher & reliable socket

fix epic failing test

remove dispatcher and reliable:
- still integration test failing due to lack of
  support for SCMP handling and more

comments, leftovers, small fixes

more minor

after rebasing

pass

add stateless dispatcher

intermediate commit

- Still things missing, e.g., update topology to include stateless dispatcher

update topology with reduced dispatcher config

add error handling and debug verbose to endHost resolution in BR

add reduced forwarding dispatcher

integration and utils

integration tests

lint + chown container

fix docker check, only for linux dev

lint

modify HP and tests

lint

fix broken rebase

lint

pass

change dispatcher configuration

bugfix: QUIC address for client with :0 port

slayers: unmap IPv4-mapped IPv6 addresses (scionproto#4377)

The Go standard library can produce IPv4-mapped IPv6 addresses when
resolving IP addresses. These IP addresses need to be unmapped before
putting them on the wire.

Before this patch, we could observe the following with tshark:

    Len=1304 SCION 1-ff00:0:110,[::ffff:172.20.2.2] -> 1-ff00:0:111,[::ffff:172.20.3.2] UDP 32769 -> 32768 1208

The regression was introduced in scionproto#4346, which removed the unmapping behavior
in slayers.PackAddr. This patch restores the behavior to ensure only
unmapped IPv4 addresses make it on the wire.

Handling the unmapping in the code that generates the addresses and only
checking this in slayers would seem ideal, but these calls are often
very far away from the place that would then trigger an error. Thus
handling this in slayers seems like a compromise that saves us and the
users of the slayers package a lot of trouble.

add packet reflection safeguard in shim

add compatible IP_PKTINFO code for windows

fix validateNextHopAddr

fix isSCMPInfo

add endhost port range configuration

port range

comment udpportRange

add fixme

allow unspecified address fon SCIONNetwork.Listen

retriving nextHop from path and local Interface information

add dispatching logic for SCMP at BR

remove dispatcher shim support for Windows

remove br dispatcher configuration from integration tests

add test for old and new br configuration with(out) shim dispatcher

comments and minor fixes

remove utils_chown container

remove docker utils from script

pass

pass

pass refactor topology endhost_port_range

comment for router

fix dispatcherless docker and integration tests

ignore SCMP errors messages on initSvcRedirect()

adapt HP test

adapt integration tests

error string HP control/main

dispatcher pass return addresses in helper function by value

fix rebase

upgrade dispatcher shim config to toml v2

add PortRange() RPC in daemon

Revert "modify HP and tests"

This reverts commit 1c82e9c.

remove leftover CSResolver leftover in HP discovery

open a single underlay socket for both the QUIC server and the SVC redirector

revert acceptance/hiden_paths test

await connectivity in old_br acceptance test

pass

pass

pass

pass

pass + lint

pass

changes to snet API + refactor

pass + allow for using snet outside the defined port range

changes in isShimDispatcher()

add destination safeguard to snet.scionConnReader.read()

add TODOs

lint

change dispatched_ports name in topo

add dispatched_ports all|ALL option

range for services in topology PortGenerator

dynamic ports refactoring

add isDispatcher flag

fix clientNet SCMPHandler

add default value for shim underlay addr

fix dispatcher port + cleaning isShimDispatcher

add dstPort check reader

remove leftover + TODO

revert destination type in ResolverPacketConn

replace UnderlayAddr

comment

comments + TODOs + refactoring

add options pattern NewCookedConn

improve error message

pass

fix rebase

rename dispatcher flag

mocks

pass

update sig_short_exp_time docker file

fix dialer constructor

fix docker image references for sig

adapt end2end test to use Dial/Listen API

remove debug logs

add comment for snet.Dial

typo

.golangcilint.yml

@@ -63,8 +63,3 @@ issues:
     - path: pkg/scrypto/cms
       linters: [goheader]
 
-    # Exceptions to errcheck for some old-ish convey tests.
-    - linters: [errcheck]
-      path: "^pkg/sock/reliable/reconnect/conn_io_test.go$|\
-             ^pkg/sock/reliable/reconnect/network_test.go$|\
-             ^pkg/sock/reliable/reconnect/reconnecter_test.go$"

acceptance/app_vs_endhost_br_dispatch/BUILD.bazel

@@ -0,0 +1,9 @@
+load("//acceptance/common:topogen.bzl", "topogen_test")
+
+topogen_test(
+    name = "test",
+    src = "test.py",
+    args = ["--executable=end2end_integration:$(location //tools/end2end_integration)"],
+    data = ["//tools/end2end_integration"],
+    topo = "//acceptance/app_vs_endhost_br_dispatch/testdata:topology.topo",
+)

acceptance/app_vs_endhost_br_dispatch/test.py

@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+
+# Copyright 2023 ETH Zurich
+
+from acceptance.common import base
+from acceptance.common import scion
+
+
+class Test(base.TestTopogen):
+    """
+    Constructs a simple test topology with one core, two leaf ASes.
+    Each of them will run a different mix between BR that will replicate
+    the legacy endhost-port-dispatch behaviour (i.e., they will send
+    traffic to its own AS to the endhost default port) and
+    application-port-dispatch routers (i.e., they will rewrite the underlay
+    UDP/IP destination port with the UDP/SCION port).
+
+    AS 1-ff00:0:1 is core.
+    AS 1-ff00:0:2, 1-ff00:0:3 are leaves.
+
+    We use the shortnames AS1, AS2, etc. for the ASes above.
+
+    AS1 contains a BR with the port rewriting configuration to the default
+    range. It also includes a shim dispatcher.
+    AS2 contains a BR with a configuration that imitates the old
+    behaviour, i.e., sending all traffic to default endhost port 30041.
+    It also includes a shim dispatcher.
+    AS3 contains a BR with the port rewriting configuration to the default
+    range. It does not include the shim dispatcher.
+    """
+
+    def setup_prepare(self):
+        super().setup_prepare()
+
+        br_as_2_id = "br1-ff00_0_2-1"
+
+        br_as_2_file = self.artifacts / "gen" / "ASff00_0_2" \
+            / ("%s.toml" % br_as_2_id)
+        scion.update_toml({"router.dispatched_port_start": 0,
+                          "router.dispatched_port_end": 0},
+                          [br_as_2_file])
+
+    def setup_start(self):
+        super().setup_start()
+        self.await_connectivity()
+
+    def _run(self):
+        ping_test = self.get_executable("end2end_integration")
+        ping_test["-d", "-outDir", self.artifacts].run_fg()
+
+
+if __name__ == "__main__":
+    base.main(Test)

acceptance/app_vs_endhost_br_dispatch/testdata/BUILD.bazel

@@ -0,0 +1,3 @@
+exports_files([
+    "topology.topo",
+])

acceptance/app_vs_endhost_br_dispatch/testdata/topology.topo

@@ -0,0 +1,15 @@
+--- # Test Topology
+ASes:
+  "1-ff00:0:1":
+    core: true
+    voting: true
+    authoritative: true
+    issuing: true
+  "1-ff00:0:2":
+    cert_issuer: 1-ff00:0:1
+  "1-ff00:0:3":
+    cert_issuer: 1-ff00:0:1
+    test_dispatcher: False
+links:
+  - {a: "1-ff00:0:1#2", b: "1-ff00:0:2#1", linkAtoB: CHILD}
+  - {a: "1-ff00:0:1#3", b: "1-ff00:0:3#1", linkAtoB: CHILD}

acceptance/common/docker.py

@@ -61,8 +61,9 @@ def collect_logs(self, out_dir: str = "logs/docker"):
         for svc in self("config", "--services").splitlines():
             # Collect logs.
             dst_f = out_p / "%s.log" % svc
+            print(svc)
             with open(dst_f, "w") as log_file:
-                cmd.docker.run(args=("logs", svc), stdout=log_file,
+                cmd.docker.run(args=("logs", "scion-"+svc+"-1"), stdout=log_file,
                                stderr=subprocess.STDOUT, retcode=None)
             # Collect coredupms.
             coredump_f = out_p / "%s.coredump" % svc

acceptance/router_benchmark/conf/topology.json

@@ -4,6 +4,7 @@
   ],
   "isd_as": "1-ff00:0:1",
   "mtu": 1400,
+  "dispatched_ports": "1024-65535",
   "border_routers": {
     "br1a": {
       "internal_addr": "10.123.10.1:30042",

acceptance/router_multi/conf/topology.json

@@ -2,6 +2,7 @@
   "isd_as": "1-ff00:0:1",
   "mtu": 1472,
   "attributes": [],
+  "dispatched_ports": "1024-65535",
   "border_routers": {
     "brA": {
       "internal_addr": "192.168.0.11:30001",

acceptance/sig_short_exp_time/BUILD.bazel

@@ -4,7 +4,6 @@ sh_test(
     srcs = ["test"],
     data = [
         "docker-compose.yml",
-        "//docker:dispatcher.tarball",
         "//docker:gateway.tarball",
         "//tools/udpproxy:udpproxy.tarball",
     ] + glob(["testdata/**"]),

acceptance/sig_short_exp_time/docker-compose.yml

@@ -47,7 +47,6 @@ services:
       bridge1:
         ipv4_address: 242.254.100.2
     volumes:
-    - vol_scion_disp_sig1-ff00_0_110:/run/shm/dispatcher:rw
     - ./testdata/1-ff00_0_110/dispatcher:/etc/scion/
     command: [ "--config", "/etc/scion/disp.toml" ]
   dispatcher2:
@@ -56,45 +55,46 @@ services:
       bridge2:
         ipv4_address: 242.254.200.2
     volumes:
-    - vol_scion_disp_sig1-ff00_0_111:/run/shm/dispatcher:rw
     - ./testdata/1-ff00_0_111/dispatcher:/etc/scion/
     command: [ "--config", "/etc/scion/disp.toml" ]
   sig1:
     cap_add:
     - NET_ADMIN
-    depends_on:
-    - dispatcher1
+    container_name: sig1
     image: scion/gateway:latest
-    network_mode: service:dispatcher1
+    networks:
+      bridge1:
+        ipv4_address: 242.254.100.2
     privileged: true
     volumes:
-    - vol_scion_disp_sig1-ff00_0_110:/run/shm/dispatcher:rw
     - /dev/net/tun:/dev/net/tun
     - ./testdata/1-ff00_0_110/sig:/etc/scion/
     command: [ "--config", "/etc/scion/sig.toml" ]
   sig2:
     cap_add:
     - NET_ADMIN
-    depends_on:
-    - dispatcher2
+    container_name: sig2
     image: scion/gateway:latest
-    network_mode: service:dispatcher2
+    networks:
+      bridge2:
+        ipv4_address: 242.254.200.2
     privileged: true
     volumes:
-    - vol_scion_disp_sig1-ff00_0_111:/run/shm/dispatcher:rw
     - /dev/net/tun:/dev/net/tun
     - ./testdata/1-ff00_0_111/sig:/etc/scion/
     command: [ "--config", "/etc/scion/sig.toml" ]
   tester1:
     image: alpine
-    network_mode: service:dispatcher1
+    networks:
+      bridge1:
+        ipv4_address: 242.254.100.10
     privileged: true
   tester2:
     image: alpine
-    network_mode: service:dispatcher2
+    networks:
+      bridge2:
+        ipv4_address: 242.254.200.10
     privileged: true
 version: '2.4'
 volumes:
   vol_logs: null
-  vol_scion_disp_sig1-ff00_0_110: null
-  vol_scion_disp_sig1-ff00_0_111: null

acceptance/sig_short_exp_time/test

@@ -44,7 +44,7 @@
 #     |                                                     |
 #     |   +---------------------------------------------+   |
 #     +---+                    pathb                    +----
-#         | 242.254.100.3:50000 <-> 242.254.200.4:50000 |
+#         | 242.254.100.4:50000 <-> 242.254.200.4:50000 |
 #         +---------------------------------------------+
 
 run_test() {(set -e

acceptance/topo_common/topology.json

@@ -1,6 +1,7 @@
 {
   "isd_as": "1-ff00:0:110",
   "mtu": 1400,
+  "dispatched_ports": "1024-65535",
   "attributes": [
     "core"
   ],

acceptance/topo_cs_reload/BUILD.bazel

@@ -15,7 +15,6 @@ go_test(
         "docker-compose.yml",
         "testdata/topology_reload.json",
         ":control.tar",
-        ":dispatcher.tar",
         ":invalid_changed_ip",
         ":invalid_changed_port",
         ":testdata/gen_crypto.sh",
@@ -36,32 +35,6 @@ go_test(
     ],
 )
 
-# dispatcher container
-oci_tarball(
-    name = "dispatcher.tar",
-    format = "docker",
-    image = ":dispatcher_image",
-    repo_tags = ["scion/" + package_name() + ":dispatcher"],
-)
-
-oci_image(
-    name = "dispatcher_image",
-    base = "//docker:dispatcher",
-    cmd = [
-        "--config",
-        "/disp.toml",
-    ],
-    entrypoint = ["/app/dispatcher"],
-    tars = [
-        ":dispatcher_data",
-    ],
-)
-
-pkg_tar(
-    name = "dispatcher_data",
-    srcs = ["testdata/disp.toml"],
-)
-
 # control container
 oci_tarball(
     name = "control.tar",

acceptance/topo_cs_reload/docker-compose.yml

@@ -7,23 +7,13 @@ networks:
       config:
       - subnet: 242.253.100.0/24
 services:
-  topo_cs_reload_dispatcher:
-    image: scion/acceptance/topo_cs_reload:dispatcher
-    networks:
-      bridge1:
-        ipv4_address: 242.253.100.2
-    volumes:
-    - vol_topo_cs_reload_disp:/run/shm/dispatcher:rw
   topo_cs_reload_control_srv:
     image: scion/acceptance/topo_cs_reload:control
-    depends_on:
-      - topo_cs_reload_dispatcher
     volumes:
-    - vol_topo_cs_reload_disp:/run/shm/dispatcher:ro
     - "${TOPO_CS_RELOAD_CONFIG_DIR}/certs:/certs:ro"
     - "${TOPO_CS_RELOAD_CONFIG_DIR}/keys:/keys:ro"
     - "${TOPO_CS_RELOAD_CONFIG_DIR}/crypto:/crypto:ro"
-    network_mode: service:topo_cs_reload_dispatcher
+    networks:
+      bridge1:
+        ipv4_address: 242.253.100.2
 version: '2.4'
-volumes:
-  vol_topo_cs_reload_disp: null

acceptance/topo_cs_reload/reload_test.go

@@ -100,10 +100,6 @@ func setupTest(t *testing.T) testState {
 	s.mustExec(t, "tar", "-xf", "crypto.tar", "-C", tmpDir)
 	// first load the docker images from bazel into the docker deamon, the
 	// tars are in the same folder as this test runs in bazel.
-	s.mustExec(t, "docker", "image", "load", "-i", "dispatcher.tar/tarball.tar")
-	t.Cleanup(func() {
-		s.mustExec(t, "docker", "image", "rm", "scion/acceptance/topo_cs_reload:dispatcher")
-	})
 	s.mustExec(t, "docker", "image", "load", "-i", "control.tar/tarball.tar")
 	t.Cleanup(func() {
 		s.mustExec(t, "docker", "image", "rm", "scion/acceptance/topo_cs_reload:control")
@@ -126,7 +122,6 @@ func (s testState) collectLogs(t *testing.T) {
 	require.NoError(t, os.MkdirAll(fmt.Sprintf("%s/logs", outdir), os.ModePerm|os.ModeDir))
 	// collect logs
 	for service, file := range map[string]string{
-		"topo_cs_reload_dispatcher":  "disp.log",
 		"topo_cs_reload_control_srv": "control.log",
 	} {
 		cmd := exec.Command("docker", "compose",

acceptance/topo_cs_reload/testdata/cs.toml

@@ -1,5 +1,4 @@
 [general]
-reconnect_to_dispatcher = true
 config_dir = "/"
 id = "cs1-ff00_0_110-1"
 

acceptance/topo_cs_reload/testdata/sd.toml

@@ -1,5 +1,4 @@
 [general]
-reconnect_to_dispatcher = true
 config_dir = "/"
 id = "sd1-ff00_0_110"
 

acceptance/topo_cs_reload/testdata/topology_reload.json

@@ -1,6 +1,7 @@
 {
   "isd_as": "1-ff00:0:110",
   "mtu": 1400,
+  "dispatched_ports": "1024-65535",
   "attributes": [
     "core"
   ],

acceptance/topo_daemon_reload/BUILD.bazel

@@ -8,7 +8,6 @@ go_test(
     data = [
         "testdata/topology_reload.json",
         ":daemon.tar",
-        ":dispatcher.tar",
         ":docker-compose.yml",
         "//acceptance/topo_common:invalid_reloads",
         "//acceptance/topo_common:topology",
@@ -24,32 +23,6 @@ go_test(
     ],
 )
 
-# dispatcher container
-oci_tarball(
-    name = "dispatcher.tar",
-    format = "docker",
-    image = ":dispatcher_image",
-    repo_tags = ["scion/" + package_name() + ":dispatcher"],
-)
-
-oci_image(
-    name = "dispatcher_image",
-    base = "//docker:dispatcher",
-    cmd = [
-        "--config",
-        "/disp.toml",
-    ],
-    entrypoint = ["/app/dispatcher"],
-    tars = [
-        ":dispatcher_data",
-    ],
-)
-
-pkg_tar(
-    name = "dispatcher_data",
-    srcs = ["testdata/disp.toml"],
-)
-
 # daemon container
 oci_tarball(
     name = "daemon.tar",

acceptance/topo_daemon_reload/docker-compose.yml

@@ -7,22 +7,14 @@ networks:
       config:
       - subnet: 242.254.100.0/24
 services:
-  topo_daemon_reload_dispatcher:
-    container_name: topo_daemon_reload_dispatcher
-    image: scion/acceptance/topo_daemon_reload:dispatcher
-    networks:
-      bridge1:
-        ipv4_address: 242.254.100.2
-    volumes:
-    - vol_topo_daemon_reload_disp:/run/shm/dispatcher:rw
   topo_daemon_reload_daemon:
     container_name: topo_daemon_reload_daemon
     image: scion/acceptance/topo_daemon_reload:daemon
     volumes:
-    - vol_topo_daemon_reload_disp:/run/shm/dispatcher:ro
     - vol_topo_daemon_reload_certs:/certs:ro
-    network_mode: service:topo_daemon_reload_dispatcher
+    networks:
+      bridge1:
+        ipv4_address: 242.254.100.2
 version: '2.4'
 volumes:
-  vol_topo_daemon_reload_disp: null
   vol_topo_daemon_reload_certs: null