api: socket options shouldn't set SO_REUSEPORT on listener socket

Currently, the socket option `SO_REUSEPORT` is always set on all
sockets - including the listener socket itself.

This doesn't take into account, that the Cilium Agent disables
port reuse on the listener socket via Envoy listener configuration
(`enable_reuse_port=false`) if BPF TPROXY is enabled.

In these cases, the cilium socket options
overrule and set the socket option `SO_REUSEPORT` - regardless of
what's configured via Envoy listener API.. This leads to
issues with BPF TPROXY (socket lookup fails).

Therefore, this commit changes the logic that the socket option
`SO_REUSEPORT` is never set on the listener socket itself.

Note: The goal of #315 was to move privileged logic into the starter
and remove the privileges from the Envoy process. But setting the socket
option `SO_REUSEPORT` isn't a privileged operation. Thus it's ok to
keep it configured via Envoy listener API.

Fixes: #315

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>

cilium/socket_option.h

@@ -84,13 +84,15 @@ class SocketMarkOption : public Network::Socket::Option,
         source_address = ipv4_source_address_;
       }
     }
+
+    uint32_t one = 1;
+
     // identity is zero for the listener socket itself, set transparent and reuse options also for
     // the listener socket.
     if (source_address || identity_ == 0) {
       // Allow reuse of the original source address. This may by needed for
       // retries to not fail on "address already in use" when using a specific
       // source address and port.
-      uint32_t one = 1;
 
       // Set ip transparent option based on the socket address family
       if (*ipVersion == Network::Address::IpVersion::v4) {
@@ -135,6 +137,13 @@ class SocketMarkOption : public Network::Socket::Option,
                   Envoy::errorDetails(status.errno_));
         return false;
       }
+    }
+
+    if (identity_ != 0) {
+      // Set SO_REUSEPORT socket option for forwarded client connections.
+      // The same option on the listener socket is controlled via the Envoy Listener option
+      // enable_reuse_port.
+      ENVOY_LOG(trace, "Set socket ({}) option SO_REUSEPORT", socket.ioHandle().fdDoNotUse());
       status = socket.setSocketOption(SOL_SOCKET, SO_REUSEPORT, &one, sizeof(one));
       if (status.return_value_ < 0) {
         ENVOY_LOG(critical, "Failed to set socket option SO_REUSEPORT: {}",