add http-response configuration keys

Add option to overwrite all the response payloads issued by haproxy, or configured by haproxy ingress. This update might lead to backward incompatibility if a deployment already customize the Lua script with the haproxy ingress generated responses. We're using two approaches here. One of them is implementing services in Lua, this gives some flexibility to issue codes that haproxy doesn't support like 495. The other approach is using HAProxy errorfile, which is the proper way to overwrite responses generated internally by HAProxy and hard to move to a Lua script, like 500 or 503. All of them however uses the same interface, so users don't need to bother if the overwrite will be made via errorfile or Lua script. The implementation was also made in such a way that a response code can be changed, this is useful in scenarios that the client should be redirected via `302 Found` if there isn't a server available (503) or if the request is denied (403).
jcmoraisjr · Apr 5, 2022 · 466ca01 · 466ca01
1 parent d7e8efd
commit 466ca01
Show file tree

Hide file tree

Showing 15 changed files with 912 additions and 80 deletions.
diff --git a/docs/content/en/docs/configuration/keys.md b/docs/content/en/docs/configuration/keys.md
@@ -391,6 +391,8 @@ The table below describes all supported configuration keys.
 | [`hsts-preload`](#hsts)                              | [true\|false]                           | Path    | `false`            |
 | [`http-log-format`](#log-format)                     | http log format                         | Global  | HAProxy default log format |
 | [`http-port`](#bind-port)                            | port number                             | Global  | `80`               |
+| [`http-response-<code>`](#http-response)             | response output                         | Global  |                    |
+| [`http-response-prometheus-root`](#http-response)    | response output                         | Global  |                    |
 | [`https-log-format`](#log-format)                    | https(tcp) log format\|`default`        | Global  | do not log         |
 | [`https-port`](#bind-port)                           | port number                             | Global  | `443`              |
 | [`https-to-http-port`](#fronting-proxy-port)         | port number                             | Global  | 0 (do not listen)  |
@@ -1673,6 +1675,124 @@ See also:
 
 ---
 
+## HTTP Response
+
+| Configuration key               | Scope    | Default | Since |
+|---------------------------------|----------|---------|-------|
+| `http-response-<code>`          | `Global` |         | v0.14 |
+| `http-response-prometheus-root` | `Global` |         | v0.14 |
+
+Overwrites the default response payload for all the HAProxy's generated HTTP responses.
+
+* `http-response-<code>`: represents all the payload of HAProxy or HAProxy Ingress generated HTTP responses. Change `<code>` to one of the supported HTTP status code, see Supported codes below.
+* `http-response-prometheus-root`: response used on requests sent to the root context of the prometheus exporter port.
+
+**Supported codes**
+
+The following list has all the HTTP status codes supported by the controller:
+
+{{% alert title="Note" %}}
+All the overwrites refer to HAProxy or HAProxy Ingress generated responses, e.g. a 403 response overwrite will not change a 403 response generated by a backend server, but instead only 403 responses that HAProxy generates itself, such as when an allow list rule denies a request to reach a backend server.
+{{% /alert %}}
+
+> All descriptions with `[haproxy]` refers to internal HAProxy responses, described in the [HAProxy documentation](https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#1.3.1) or in the [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status). All the others are handled and issued by HAProxy Ingress configurations.
+
+| Code  | Reason | Description |
+|-------|--------|-------------|
+| `200` | OK | `[haproxy]` |
+| `400` | Bad Request | `[haproxy]` |
+| `401` | Unauthorized | `[haproxy]` |
+| `403` | Forbidden | `[haproxy]` |
+| `404` | Not Found | The requested host and path was not found, the `--default-backend-service` command-line is not used and there is no ingress configured as the default backend with a matching path. |
+| `405` | Method Not Allowed | `[haproxy]` |
+| `407` | Proxy Authentication Required | `[haproxy]` |
+| `408` | Request Timeout | `[haproxy]` |
+| `410` | Gone | `[haproxy]` |
+| `413` | Payload Too Large | A request is bigger than specified in the `proxy-body-size` configuration key. |
+| `421` | Misdirected Request | Incoming SNI was used to match a hostname and the Host header has a distinct value. |
+| `425` | Too Early | `[haproxy]` |
+| `429` | Too Many Requests | `[haproxy]` |
+| `495` | SSL Certificate Error | An invalid certificate was used on a mTLS connection. |
+| `496` | SSL Certificate Required | A certificate wasn't used on a mTLS connection but a certificate is mandatory. |
+| `500` | Internal Server Error | `[haproxy]` |
+| `501` | Not Implemented | `[haproxy]` |
+| `502` | Bad Gateway | `[haproxy]` |
+| `503` | Service Unavailable | `[haproxy]` |
+| `504` | Gateway Timeout | `[haproxy]` |
+
+**Syntax**
+
+A multi-line configuration is used to customize the response payload on all the configuration keys:
+
+* The very first line: Optional, the HTTP status code of the response, optionally followed by the status reason used on HTTP/1.1 responses. The default value is used if missing. Valid inputs are e.g. `404` or `404 Not Found`.
+* Lines before the first empty line: Optional HTTP headers, one per line, whose name and value are separated by a colon `:`. It is recommended to always add `content-type` header. `content-length` is always calculated and should not be used.
+* Lines after the first empty line: Optional HTTP body. It will be copied verbatim to a Lua script. Any char is allowed here except the `]==]` string which is reserved by the controller.
+
+Some general hints about response overwriting:
+
+* Do not create huge responses, the whole overwrite must fit into the internal buffer, which is 16k by default, leaving some room to configured downstream rules to operate. See [HAProxy's errorfile doc](https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#4-errorfile).
+* Take care with external links, e.g. the overwrite of a 503 error page might lead to another 503 error.
+* Only add the status code line if changing the code, otherwise let the controller configure with default values.
+* A missing status code and status reason will lead to default values, but missing headers and missing body will lead to, respectively, only the `content-length` header and an empty body.
+* Always add at least the HTTP header `content-type` with the correct value.
+* The HTTP header `content-length` will be overwritten if used.
+
+**Examples**
+
+Change the payload type and content, using the default status code and status reason:
+
+```yaml
+  data:
+    http-response-404: |
+      content-type: text/plain
+      connection: close
+
+      404 not found
+```
+
+Overwrite the Status Code - always add the status reason (see [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status)) if changing the status code:
+
+```yaml
+  data:
+    http-response-404: |
+      302 Found
+      location: https://this.other.local
+```
+
+Response without header and with an empty body:
+
+```yaml
+  data:
+    http-response-404: |
+      404 Not Found
+```
+
+Response with the headers only:
+
+```yaml
+  data:
+    http-response-404: |
+      connection: close
+```
+
+Response with only the body - discouraged, at least the content-type should be added:
+
+```yaml
+  data:
+    http-response-404: |
+
+      not found
+```
+
+See also:
+
+* [`--default-backend-service`]({{% relref "command-line#default-backend-service" %}}) command-line option
+* [`proxy-body-size`](#proxy-body-size) configuration key
+* [mTLS](#auth-tls) related configuration keys
+* https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#4-errorfile
+* HAProxy's HTTP response at [HAProxy documentation](https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#1.3.1)
+* HTTP response status codes at [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status)
+
 ## Initial weight
 
 | Configuration key | Scope     | Default | Since  |

diff --git a/pkg/common/ingress/controller/launch.go b/pkg/common/ingress/controller/launch.go
@@ -434,6 +434,8 @@ tracked.`)
 		&ingress.DefaultCACertsDirectory,
 		&ingress.DefaultCrlDirectory,
 		&ingress.DefaultVarRunDirectory,
+		&ingress.DefaultErrorfilesDirectory,
+		&ingress.DefaultLuaScriptsDirectory,
 		&ingress.DefaultMapsDirectory,
 	} {
 		// TODO evolve this ugly trick to a proper struct that allows custom configuration

diff --git a/pkg/common/ingress/types.go b/pkg/common/ingress/types.go
@@ -35,7 +35,10 @@ var (
 	DefaultCACertsDirectory = "/var/lib/haproxy/cacerts"
 	DefaultCrlDirectory     = "/var/lib/haproxy/crl"
 	DefaultVarRunDirectory  = "/var/run/haproxy"
-	DefaultMapsDirectory    = "/etc/haproxy/maps"
+	//
+	DefaultErrorfilesDirectory = "/etc/haproxy/errorfiles"
+	DefaultLuaScriptsDirectory = "/etc/haproxy/lua"
+	DefaultMapsDirectory       = "/etc/haproxy/maps"
 )
 
 // Controller holds the methods to handle an Ingress backend

diff --git a/pkg/converters/ingress/annotations/global.go b/pkg/converters/ingress/annotations/global.go
@@ -443,3 +443,204 @@ func (c *updater) buildGlobalCustomConfig(d *globalData) {
 	}
 	d.global.CustomProxy = proxy
 }
+
+// TODO these defaults should be in default.go but currently ingress parsing
+// doesn't preserve the hardcoded default, overwriting it with the user's
+// provided one. We need it in the case the user input has some error.
+// Need to improve this behavior and then we can move this to the defaults.
+const (
+	httpResponse404 = `404
+Content-Type: text/html
+Cache-Control: no-cache
+
+<html><body><h1>404 Not Found</h1>
+The requested URL was not found.
+</body></html>
+`
+
+	httpResponse413 = `413
+Content-Type: text/html
+Cache-Control: no-cache
+
+<html><body><h1>413 Request Entity Too Large</h1>
+The request is too large.
+</body></html>
+`
+
+	httpResponse421 = `421
+Content-Type: text/html
+Cache-Control: no-cache
+
+<html><body><h1>421 Misdirected Request</h1>
+Request sent to a non-authoritative server.
+</body></html>
+`
+
+	httpResponse495 = `495
+Content-Type: text/html
+Cache-Control: no-cache
+
+<html><body><h1>495 SSL Certificate Error</h1>
+An invalid certificate has been provided.
+</body></html>
+`
+
+	httpResponse496 = `496
+Content-Type: text/html
+Cache-Control: no-cache
+
+<html><body><h1>496 SSL Certificate Required</h1>
+A client certificate must be provided.
+</body></html>
+`
+
+	httpResponsePrometheusRoot = `200
+Content-Type: text/html
+Cache-Control: no-cache
+
+<html>
+<head><title>HAProxy Exporter</title></head>
+<body><h1>HAProxy Exporter</h1>
+<a href='/metrics'>Metrics</a>
+</body></html>
+`
+)
+
+var customHTTPResponses = []struct {
+	name   string
+	code   int
+	reason string
+	key    string
+	def    string
+}{
+	// Lua based, need name and default
+	{"send-prometheus-root", 200, "OK", ingtypes.GlobalHTTPResponsePrometheusRoot, httpResponsePrometheusRoot},
+	{"send-404", 404, "Not Found", ingtypes.GlobalHTTPResponse404, httpResponse404},
+	{"send-413", 413, "Payload Too Large", ingtypes.GlobalHTTPResponse413, httpResponse413},
+	{"send-421", 421, "Misdirected Request", ingtypes.GlobalHTTPResponse421, httpResponse421},
+	{"send-495", 495, "SSL Certificate Error", ingtypes.GlobalHTTPResponse495, httpResponse495},
+	{"send-496", 496, "SSL Certificate Required", ingtypes.GlobalHTTPResponse496, httpResponse496},
+	// HAProxy based, default isn't used because the response will be ignored if conf is missing
+	// This configuration assumes that:
+	//  - `name` will be used as the internal status code, `code` is the real status that should be returned
+	//  - `def` should always be empty, this is used to distinguish between Lua and HAProxy based config
+	{"200", 200, "OK", ingtypes.GlobalHTTPResponse200, ""},
+	{"400", 400, "Bad Request", ingtypes.GlobalHTTPResponse400, ""},
+	{"401", 401, "Unauthorized", ingtypes.GlobalHTTPResponse401, ""},
+	{"403", 403, "Forbidden", ingtypes.GlobalHTTPResponse403, ""},
+	{"405", 405, "Method Not Allowed", ingtypes.GlobalHTTPResponse405, ""},
+	{"407", 407, "Proxy Authentication Required", ingtypes.GlobalHTTPResponse407, ""},
+	{"408", 408, "Request Timeout", ingtypes.GlobalHTTPResponse408, ""},
+	{"410", 410, "Gone", ingtypes.GlobalHTTPResponse410, ""},
+	{"425", 425, "Too Early", ingtypes.GlobalHTTPResponse425, ""},
+	{"429", 429, "Too Many Requests", ingtypes.GlobalHTTPResponse429, ""},
+	{"500", 500, "Internal Server Error", ingtypes.GlobalHTTPResponse500, ""},
+	{"501", 501, "Not Implemented", ingtypes.GlobalHTTPResponse501, ""},
+	{"502", 502, "Bad Gateway", ingtypes.GlobalHTTPResponse502, ""},
+	{"503", 503, "Service Unavailable", ingtypes.GlobalHTTPResponse503, ""},
+	{"504", 504, "Gateway Timeout", ingtypes.GlobalHTTPResponse504, ""},
+}
+
+func (c *updater) buildGlobalCustomResponses(d *globalData) {
+	var luaResponses []hatypes.HTTPResponse
+	var haResponses []hatypes.HTTPResponse
+	for _, data := range customHTTPResponses {
+		var response *hatypes.HTTPResponse
+		var err error
+		if content := d.mapper.Get(data.key).Value; content != "" {
+			response, err = parseHeadAndBody(content)
+			if err != nil {
+				c.logger.Warn("ignoring '%s' due to a malformed response: %v", data.key, err)
+			}
+		}
+		if data.def != "" {
+			// there is a default value, so a valid response should
+			// always be provided -- used by Lua script based responses
+			if response == nil || err != nil {
+				response, err = parseHeadAndBody(data.def)
+			}
+			if err != nil {
+				// this means that the default is broken
+				panic(err)
+			}
+		}
+		if response == nil {
+			// response is optional and wasn't created, just skip
+			continue
+		}
+		response.Name = data.name
+		if response.StatusCode == 0 {
+			response.StatusCode = data.code
+		}
+		if response.StatusReason == "" {
+			response.StatusReason = data.reason
+		}
+		if data.def == "" {
+			// this is currently the simplest way to distinguish between
+			// Lua and HAProxy based responses
+			haResponses = append(haResponses, *response)
+		} else {
+			luaResponses = append(luaResponses, *response)
+		}
+	}
+	d.global.CustomHTTPLuaResponses = luaResponses
+	d.global.CustomHTTPHAResponses = haResponses
+}
+
+var statusCodeRegex = regexp.MustCompile(`^([0-9]{3})( [A-Za-z ]+)?$`)
+
+func parseHeadAndBody(content string) (*hatypes.HTTPResponse, error) {
+	body := false
+	bodysize := 0
+	response := &hatypes.HTTPResponse{}
+	response.Headers = []hatypes.HTTPHeader{{Name: "Content-Length"}}
+	for i, line := range utils.LineToSlice(content) {
+		line = strings.TrimRight(line, " ")
+		if i == 0 && statusCodeRegex.MatchString(line) {
+			// very first line with status code pattern
+			status := statusCodeRegex.FindStringSubmatch(line)
+			code, _ := strconv.Atoi(status[1])
+			if code < 101 || code > 599 {
+				return nil, fmt.Errorf("err")
+			}
+			response.StatusCode = code
+			response.StatusReason = strings.TrimSpace(status[2])
+		} else if line == "" {
+			// no more headers
+			body = true
+		} else if !body {
+			// header
+			pos := strings.Index(line, ":")
+			if pos < 0 {
+				return nil, fmt.Errorf("missing a colon ':' in the header declaration: %s", line)
+			}
+			header := hatypes.HTTPHeader{
+				Name:  strings.TrimSpace(line[:pos]),
+				Value: strings.TrimSpace(line[pos+1:]),
+			}
+			if strings.ToLower(header.Name) == "content-length" {
+				// we will overwrite anything the user tried to add
+				continue
+			}
+			if strings.ContainsAny(header.Name, `" `) {
+				return nil, fmt.Errorf("invalid chars in the header name: '%s'", header.Name)
+			}
+			if header.Name == "" || header.Value == "" {
+				return nil, fmt.Errorf("header name and value must not be empty: '%s'", line)
+			}
+			if strings.ContainsAny(header.Value, `"`) {
+				return nil, fmt.Errorf("invalid chars in the header value: '%s'", header.Value)
+			}
+			response.Headers = append(response.Headers, header)
+		} else {
+			// body
+			if strings.Contains(line, "]==]") {
+				return nil, fmt.Errorf("the string ']==]' cannot be used in the body")
+			}
+			response.Body = append(response.Body, line)
+			bodysize += len(line) + 1 // length of the line as an array of bytes, plus a unix line break
+		}
+	}
+	response.Headers[0].Value = strconv.Itoa(bodysize)
+	return response, nil
+}