From 3fee2aa9356223b68b1c69c6bf1ab26479521bc4 Mon Sep 17 00:00:00 2001 From: Matt Leung Date: Wed, 10 Mar 2021 16:50:01 -0800 Subject: [PATCH] Add documentation for adding CIDRs to the failsafe rules (#4461) * Add documentation for adding CIDRs to the failsafe rules * update crd --- ...projectcalico.org_felixconfigurations.yaml | 52 +++++++++++-------- reference/felix/configuration.md | 4 +- reference/resources/felixconfig.md | 13 ++--- security/protect-hosts.md | 26 +++++----- 4 files changed, 53 insertions(+), 42 deletions(-) diff --git a/_includes/charts/calico/crds/kdd/crd.projectcalico.org_felixconfigurations.yaml b/_includes/charts/calico/crds/kdd/crd.projectcalico.org_felixconfigurations.yaml index 62a193d74f6..8557ab03800 100644 --- a/_includes/charts/calico/crds/kdd/crd.projectcalico.org_felixconfigurations.yaml +++ b/_includes/charts/calico/crds/kdd/crd.projectcalico.org_felixconfigurations.yaml @@ -166,48 +166,58 @@ spec: type: array failsafeInboundHostPorts: description: 'FailsafeInboundHostPorts is a comma-delimited list of - UDP/TCP ports that Felix will allow incoming traffic to host endpoints - on irrespective of the security policy. This is useful to avoid - accidentally cutting off a host with incorrect configuration. Each - port should be specified as tcp: or udp:. + UDP/TCP ports and CIDRs that Felix will allow incoming traffic to + host endpoints on irrespective of the security policy. This is useful + to avoid accidentally cutting off a host with incorrect configuration. + Each port should be specified as tcp:: or udp::. For back-compatibility, if the protocol is not specified, it defaults - to "tcp". To disable all inbound host ports, use the value none. - The default value allows ssh access and DHCP. [Default: tcp:22, - udp:68, tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667]' + to "tcp". If a CIDR is not specified, it will default to `0.0.0.0/0`. + To disable all inbound host ports, use the value none. The default + value allows ssh access and DHCP. [Default: tcp:0.0.0.0/0:22, udp:0.0.0.0/0:68, + tcp:0.0.0.0/0:179, tcp:0.0.0.0/0:2379, tcp:0.0.0.0/0:2380, tcp:0.0.0.0/0:6443, + tcp:0.0.0.0/0:6666, tcp:0.0.0.0/0:6667]' items: - description: ProtoPort is combination of protocol and port, both - must be specified. + description: ProtoPort is combination of protocol, port, and CIDR. + All three must be specified. properties: + net: + type: string port: type: integer protocol: type: string required: + - net - port - protocol type: object type: array failsafeOutboundHostPorts: description: 'FailsafeOutboundHostPorts is a comma-delimited list - of UDP/TCP ports that Felix will allow outgoing traffic from host - endpoints to irrespective of the security policy. This is useful - to avoid accidentally cutting off a host with incorrect configuration. - Each port should be specified as tcp: or udp:. - For back-compatibility, if the protocol is not specified, it defaults - to "tcp". To disable all outbound host ports, use the value none. - The default value opens etcd''s standard ports to ensure that Felix - does not get cut off from etcd as well as allowing DHCP and DNS. - [Default: tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667, - udp:53, udp:67]' + of UDP/TCP ports and CIDRs that Felix will allow outgoing traffic + from host endpoints to irrespective of the security policy. This + is useful to avoid accidentally cutting off a host with incorrect + configuration. Each port should be specified as tcp: + or udp:. For back-compatibility, if the protocol is + not specified, it defaults to "tcp". If a CIDR is not specified, + it will default to `0.0.0.0/0`. To disable all outbound host ports, + use the value none. The default value opens etcd''s standard ports + to ensure that Felix does not get cut off from etcd as well as allowing + DHCP and DNS. [Default: tcp:0.0.0.0/0:179, tcp:0.0.0.0/0:2379, tcp:0.0.0.0/0:2380, + tcp:0.0.0.0/0:6443, tcp:0.0.0.0/0:6666, tcp:0.0.0.0/0:6667, udp:0.0.0.0/0:53, + udp:0.0.0.0/0:67]' items: - description: ProtoPort is combination of protocol and port, both - must be specified. + description: ProtoPort is combination of protocol, port, and CIDR. + All three must be specified. properties: + net: + type: string port: type: integer protocol: type: string required: + - net - port - protocol type: object diff --git a/reference/felix/configuration.md b/reference/felix/configuration.md index 296cb51a8e3..153795f3d34 100644 --- a/reference/felix/configuration.md +++ b/reference/felix/configuration.md @@ -45,8 +45,8 @@ The full list of parameters which can be set is as follows. | `EndpointReportingDelaySecs` | `FELIX_ENDPOINTREPORTINGDELAYSECS` | Set the endpoint reporting delay between status check intervals, in seconds. Only used if endpoint reporting is enabled. [Default: `1`] | int ] | `EndpointReportingEnabled` | `FELIX_ENDPOINTREPORTINGENABLED` | Enable the endpoint status reporter. [Default: `false`] | boolean | | `ExternalNodesCIDRList` | `FELIX_EXTERNALNODESCIDRLIST` | Comma-delimited list of IPv4 or CIDR of external-non-calico-nodes from which IPIP traffic is accepted by calico-nodes. [Default: ""] | string | -| `FailsafeInboundHostPorts` | `FELIX_FAILSAFEINBOUNDHOSTPORTS` | Comma-delimited list of UDP/TCP/SCTP ports that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. Each port should be specified as `tcp|udp|sctp:`. For backwards compatibility, if the protocol is not specified, it defaults to "tcp". To disable all inbound host ports, use the value `none`. The default value allows ssh access, DHCP, BGP, etcd and the Kubernetes API. [Default: `tcp:22, udp:68, tcp:179, tcp:2379, tcp:2380, tcp:5473, tcp:6443, tcp:6666, tcp:6667`] | string | -| `FailsafeOutboundHostPorts` | `FELIX_FAILSAFEOUTBOUNDHOSTPORTS` | Comma-delimited list of UDP/TCP/SCTP ports that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidently cutting off a host with incorrect configuration. Each port should be specified as `tcp|udp|sctp:`. For backwards compatibility, if the protocol is not specified, it defaults to "tcp". To disable all outbound host ports, use the value `none`. The default value opens etcd's standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP, DNS, BGP and the Kubernetes API. [Default: `udp:53, udp:67, tcp:179, tcp:2379, tcp:2380, tcp:5473, tcp:6443, tcp:6666, tcp:6667`] | string | +| `FailsafeInboundHostPorts` | `FELIX_FAILSAFEINBOUNDHOSTPORTS` | Comma-delimited list of UDP/TCP/SCTP ports and CIDRs that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. Each port should be specified as `tcp|udp|sctp::`. For backwards compatibility, if the protocol is not specified, it defaults to "tcp". If a CIDR is not specified, it will default to `0.0.0.0/0`. To disable all inbound host ports, use the value `none`. The default value allows ssh access, DHCP, BGP, etcd and the Kubernetes API. [Default: `tcp:0.0.0.0/0:22, udp:0.0.0.0/0:68, tcp:0.0.0.0/0:179, tcp:0.0.0.0/0:2379, tcp:0.0.0.0/0:2380, tcp:0.0.0.0/0:5473, tcp:0.0.0.0/0:6443, tcp:0.0.0.0/0:6666, tcp:0.0.0.0/0:6667`] | string | +| `FailsafeOutboundHostPorts` | `FELIX_FAILSAFEOUTBOUNDHOSTPORTS` | Comma-delimited list of UDP/TCP/SCTP ports and CIDRs that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidently cutting off a host with incorrect configuration. Each port should be specified as `tcp|udp|sctp::`. For backwards compatibility, if the protocol is not specified, it defaults to "tcp". If a CIDR is not specified, it will default to `0.0.0.0/0`. To disable all outbound host ports, use the value `none`. The default value opens etcd's standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP, DNS, BGP and the Kubernetes API. [Default: `udp:0.0.0.0/0:53, udp:0.0.0.0/0:67, tcp:0.0.0.0/0:179, tcp:0.0.0.0/0:2379, tcp:0.0.0.0/0:2380, tcp:0.0.0.0/0:5473, tcp:0.0.0.0/0:6443, tcp:0.0.0.0/0:6666, tcp:0.0.0.0/0:6667`] | string | | `FelixHostname` | `FELIX_FELIXHOSTNAME` | The hostname Felix reports to the plugin. Should be used if the hostname Felix autodetects is incorrect or does not match what the plugin will expect. [Default: `socket.gethostname()`] | string | | `GenericXDPEnabled` | `FELIX_GENERICXDPENABLED` | When enabled, Felix can fallback to the non-optimized `generic` XDP mode. This should only be used for testing since it doesn't improve performance over the non-XDP mode. [Default: `false`] | boolean | | `HealthEnabled` | `FELIX_HEALTHENABLED` | When enabled, exposes felix health information via an http endpoint. | boolean | diff --git a/reference/resources/felixconfig.md b/reference/resources/felixconfig.md index 325b5d35ecb..7f485713069 100644 --- a/reference/resources/felixconfig.md +++ b/reference/resources/felixconfig.md @@ -44,8 +44,8 @@ spec: | defaultEndpointToHostAction | This parameter controls what happens to traffic that goes from a workload endpoint to the host itself (after the traffic hits the endpoint egress policy). By default {{site.prodname}} blocks traffic from workload endpoints to the host itself with an iptables "DROP" action. If you want to allow some or all traffic from endpoint to host, set this parameter to `Return` or `Accept`. Use `Return` if you have your own rules in the iptables "INPUT" chain; {{site.prodname}} will insert its rules at the top of that chain, then `Return` packets to the "INPUT" chain once it has completed processing workload endpoint egress policy. Use `Accept` to unconditionally accept packets from workloads after processing workload endpoint egress policy. | Drop, Return, Accept | string | `Drop` | | deviceRouteSourceAddress | IPv4 address to set as the source hint for routes programmed by Felix. When not set the source address for local traffic from host to workload will be determined by the kernel. | IPv4 | string | `""` | | deviceRouteProtocol | This defines the route protocol added to programmed device routes. | Protocol | int | RTPROT_BOOT | -| failsafeInboundHostPorts | UDP/TCP/SCTP protocol/port pairs that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. The default value allows SSH access, etcd, BGP, DHCP and the Kubernetes API. | | List of [ProtoPort](#protoport) | {::nomarkdown}

- protocol: tcp
  port: 22
- protocol: udp
  port: 68
- protocol: tcp
  port: 179
- protocol: tcp
  port: 2379
- protocol: tcp
  port: 2380
- protocol: tcp
  port: 5473
- protocol: tcp
  port: 6443
- protocol: tcp
  port: 6666
- protocol: tcp
  port: 6667

{:/} | -| failsafeOutboundHostPorts | UDP/TCP/SCTP protocol/port pairs that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. The default value opens etcd's standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP, DNS, BGP and the Kubernetes API. | | List of [ProtoPort](#protoport) | {::nomarkdown}

- protocol: udp
  port: 53
- protocol: udp
  port: 67
- protocol: tcp
  port: 179
- protocol: tcp
  port: 2379
- protocol: tcp
  port: 2380
- protocol: tcp
  port: 5473
- protocol: tcp
  port: 6443
- protocol: tcp
  port: 6666
- protocol: tcp
  port: 6667

{:/} | +| failsafeInboundHostPorts | UDP/TCP/SCTP protocol/cidr/port groupings that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. The default value allows SSH access, etcd, BGP, DHCP and the Kubernetes API. | | List of [ProtoPort](#protoport) | {::nomarkdown}

- protocol: tcp
  net: 0.0.0.0/0
  port: 22
- protocol: udp
  net: 0.0.0.0/0
  port: 68
- protocol: tcp
  net: 0.0.0.0/0
  port: 179
- protocol: tcp
  net: 0.0.0.0/0
  port: 2379
- protocol: tcp
  net: 0.0.0.0/0
  port: 2380
- protocol: tcp
  net: 0.0.0.0/0
  port: 5473
- protocol: tcp
  net: 0.0.0.0/0
  port: 6443
- protocol: tcp
  net: 0.0.0.0/0
  port: 6666
- protocol: tcp
  net: 0.0.0.0/0
  port: 6667

{:/} | +| failsafeOutboundHostPorts | UDP/TCP/SCTP protocol/port groupings that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. The default value opens etcd's standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP, DNS, BGP and the Kubernetes API. | | List of [ProtoPort](#protoport) | {::nomarkdown}

- protocol: udp
  net: 0.0.0.0/0
  port: 53
- protocol: udp
  net: 0.0.0.0/0
  port: 67
- protocol: tcp
  net: 0.0.0.0/0
  port: 179
- protocol: tcp
  net: 0.0.0.0/0
  port: 2379
- protocol: tcp
  net: 0.0.0.0/0
  port: 2380
- protocol: tcp
  net: 0.0.0.0/0
  port: 5473
- protocol: tcp
  net: 0.0.0.0/0
  port: 6443
- protocol: tcp
  net: 0.0.0.0/0
  port: 6666
- protocol: tcp
  net: 0.0.0.0/0
  port: 6667

{:/} | | featureDetectOverride | Is used to override the feature detection. Values are specified in a comma separated list with no spaces, example; "SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=". "true" or "false" will force the feature, empty or omitted values are auto-detected. | string | string | `""` | | genericXDPEnabled | When enabled, Felix can fallback to the non-optimized `generic` XDP mode. This should only be used for testing since it doesn't improve performance over the non-XDP mode. | true,false | boolean | `false` | | interfaceExclude | A comma-separated list of interface names that should be excluded when Felix is resolving host endpoints. The default value ensures that Felix ignores Kubernetes' internal `kube-ipvs0` device. If you want to exclude multiple interface names using a single value, the list supports regular expressions. For regular expressions you must wrap the value with `/`. For example having values `/^kube/,veth1` will exclude all interfaces that begin with `kube` and also the interface `veth1`. | string | string | `kube-ipvs0` | @@ -116,10 +116,11 @@ spec: #### ProtoPort -| Field | Description | Accepted Values | Schema | -|----------|----------------------|-------------------|--------| -| port | The exact port match | 0-65535 | int | -| protocol | The protocol match | tcp, udp, sctp | string | +| Field | Description | Accepted Values | Schema | +|----------|----------------------|--------------------------------------|--------| +| port | The exact port match | 0-65535 | int | +| protocol | The protocol match | tcp, udp, sctp | string | +| net | The CIDR match | any valid CIDR (e.g. 192.168.0.0/16) | string | #### RouteTableRange diff --git a/security/protect-hosts.md b/security/protect-hosts.md index c34cc544937..38d957e607f 100644 --- a/security/protect-hosts.md +++ b/security/protect-hosts.md @@ -67,22 +67,22 @@ If you are already running {{site.prodname}} for Kubernetes, you are good to go. #### Avoid accidentally cutting all host connectivity -To avoid inadvertently cutting all host connectivity because of non-existent or misconfigured network policy, {{site.prodname}} uses failsafe rules that open specific ports on all host endpoints. +To avoid inadvertently cutting all host connectivity because of non-existent or misconfigured network policy, {{site.prodname}} uses failsafe rules that open specific ports and CIDRs on all host endpoints. Review the following table to determine if the defaults work for your implementation. If not, change the default ports using the parameters, **FailsafeInboundHostPorts** and **FailsafeOutboundHostPorts** in [Configuring Felix]({{ site.baseurl }}/reference/felix/configuration#environment-variables). -| Port | Protocol | Direction | Purpose | -|--------|----------|---------------------|------------------------------------------------| -| 22 | TCP | Inbound | SSH access | -| 53 | UDP | Outbound | DNS queries | -| 67 | UDP | Outbound | DHCP access | -| 68 | UDP | Inbound | DHCP access | -| 179 | TCP | Inbound & Outbound | BGP access ({{site.prodname}} networking) | -| 2379 | TCP | Inbound & Outbound | etcd access | -| 2380 | TCP | Inbound & Outbound | etcd access | -| 6443 | TCP | Inbound & Outbound | Kubernetes API server access | -| 6666 | TCP | Inbound & Outbound | etcd self-hosted service access | -| 6667 | TCP | Inbound & Outbound | etcd self-hosted service access | +| Port | Protocol | CIDR | Direction | Purpose | +|--------|----------|------------|---------------------|------------------------------------------------| +| 22 | TCP | 0.0.0.0/0 | Inbound | SSH access | +| 53 | UDP | 0.0.0.0/0 | Outbound | DNS queries | +| 67 | UDP | 0.0.0.0/0 | Outbound | DHCP access | +| 68 | UDP | 0.0.0.0/0 | Inbound | DHCP access | +| 179 | TCP | 0.0.0.0/0 | Inbound & Outbound | BGP access ({{site.prodname}} networking) | +| 2379 | TCP | 0.0.0.0/0 | Inbound & Outbound | etcd access | +| 2380 | TCP | 0.0.0.0/0 | Inbound & Outbound | etcd access | +| 6443 | TCP | 0.0.0.0/0 | Inbound & Outbound | Kubernetes API server access | +| 6666 | TCP | 0.0.0.0/0 | Inbound & Outbound | etcd self-hosted service access | +| 6667 | TCP | 0.0.0.0/0 | Inbound & Outbound | etcd self-hosted service access | #### Use policy to restrict host traffic