Support DSCP Marking on external interfaces for QoS purposes

[kfaseela]

Describe the feature request

Network Functions shall support both server side (ingress gateway) and client side (egress sidecar/gateway) DSCP marking for interfaces including provisioning. Customers who use 5G Core solutions need a mechanism to set DSCP marking on all external facing interfaces. So this becomes an important requirement for telco deployments. Similar request can be seen already on Contour which is being worked upon. Other socket options like TCP KeepAlive seems to be already supported in Contour.

Background

Differentiated services or DiffServ is a computer networking architecture that specifies a simple and scalable mechanism for classifying and managing network traffic. DiffServ uses a 6-bit differentiated services code point (DSCP) in the 8-bit differentiated services field (DS field) in the IP header for packet classification purposes.

EnvoyFilter based solution

Envoy proxy has already exposed socket option through its XDS API (for both listener and cluster). The envoy proxy listener socket option DSCP configuration can work for Istio ingress gateway scenario, as well as Istio egress sidecar/gateway scenario. However, EnvoyFilters are always complex to manage and maintain.

A sample EnvoyFilter for DSCP socket option modification for IPv4 and IPv6 are given below:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ingress-dscp-socketoptions
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: istio-ingressgateway
  configPatches:
  - applyTo: LISTENER
    match:
      context: GATEWAY
      listener:
        portNumber: 8443
    patch:
      operation: MERGE
      value:
        socket_options:   #  this values need to be double checked!
        - level: 0        #  0 for IPPROTO_IP
          name: 1         #  1 for IP_TOS
          int_value: 64   #0x10 IPTOS_LOWDELAY  ==> 10000(00)=2^6=64  0x08 IPTOS_THROUGHPUT ==> 1000(00)=2^5=32
          state: STATE_PREBIND  #STATE_PREBIND is default, other 2 values are STATE_BOUND, STATE_LISTENING

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ingress-dscp-socketoptions
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: istio-ingressgateway
  configPatches:
  - applyTo: LISTENER
    match:
      context: GATEWAY
      listener:
        portNumber: 8443
    patch:
      operation: MERGE
      value:
        socket_options:   #  this values need to be double checked!
        - level: 41        #  41 for IPPROTO_IPv6
          name: 67         #  67 for IPV6_TCLASS
          int_value: 32    ##IPV6_TCLASS is 32, and DSCP is 0x20, 0x10 IPTOS_LOWDELAY  ==> 10000(00)=2^6=64  0x08 IPTOS_THROUGHPUT ==> 1000(00)=2^5=32
          state: STATE_PREBIND  #STATE_PREBIND is default, other 2 values are STATE_BOUND, STATE_LISTENING

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: egress-dscp-socketoptions
spec:
  configPatches:
  - applyTo: CLUSTER
    match:
      cluster:
        name: "outbound|443||traffic.foo.com"
    patch:
      operation: MERGE
      value:
        upstream_bind_config:
          source_address:
            address: "0.0.0.0"
            port_value: 0
          socket_options:
          - level: 0        #  0 for IPPROTO_IP
            name: 1         #  1 for IP_TOS
            int_value: 32  #IP_TOS is 72, and DSCP is 0x48
            state: STATE_PREBIND  #STATE_PREBIND is default, other 2 values are STATE_BOUND, STATE_LISTENING

Solution Proposal

It would be nice if DSCP socket options configuration is supported via standard istio APIs.
For gateways, https://istio.io/latest/docs/reference/config/networking/gateway/#Port seems to be the right place for adding the same. For egress sidecar, https://istio.io/latest/docs/reference/config/networking/destination-rule/#TrafficPolicy-PortTrafficPolicy can be enhanced to add DSCP marking.

Affected product area (please put an X in all that apply)

[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

[hzxuzhonghu]

Looks reasonable requirement to me

[jacob-delgado]

There have been numerous times when we've had to modify socket options via EnvoyFilters. It'd be great to have this implemented, but the only concern I have is that you need to pass in flags/integers to set these options. I'd be fine with us passing integers rather than trying to enumerate all possible settings (near impossible).

[kfaseela]

Thanks for your feedback @hzxuzhonghu and @jacob-delgado ! @howardjohn would be good to know your feedback as well, before I start with the implementation.

[howardjohn]

My main concern is that Gateway is merged. So defining listener properties in Gateway is not reliable. The best we can do is reject when there are conflicts. This makes it dangerous and hard to configure. For example, if I have 2 Gateways to start with that are merged. Then I apply DSCP to one of them... now suddenly it will be ignored and cause an outage. Its actually impossible to enable on existing Gateways safely when there are multiple since it cannot be done atomically.

IMO it is also pretty niche. I feel EnvoyFilter may still be most appropriate here.

[kfaseela]

Based on further discussions with @howardjohn @ramaraochavali @jacob-delgado , it looks like EnvoyFilter itself would be the best way forward, as adding this to Gateway may cause additional complexities. We currently have issues on dynamic-updates of socket_options at listener level, but for that have raised issues in the envoy community.

[hobbytp]

F.Y.I.
PR for dynamic update of DSCP socket options: [WIP] listener: support update socket_options by soulxu · Pull Request #23189 · envoyproxy/envoy (github.com)