Skip to content

Commit

Permalink
topdown: Add raw_body parameter to http.send
Browse files Browse the repository at this point in the history 
Instead of changing the behaviour of the body parameter (which is
always sent as JSON) just add a new parameter that gives the caller
complete control over the message body.

Also, refactor the docs for the function to be a bit more readable.

Fixes #1903

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
  • Loading branch information
@tsandall
tsandall committed Nov 14, 2019
1 parent c19eea7 commit cdcebd8
Show file tree
Hide file tree
Showing 4 changed files with 123 additions and 60 deletions.
59 changes: 37 additions & 22 deletions docs/content/policy-reference.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -424,36 +424,51 @@ Note that the opa executable will need access to the timezone files in the envir

| Built-in | Description |
| ------- |-------------|
| <span class="opa-keep-it-together">``http.send(request, output)``</span> | ``http.send`` executes a HTTP request and returns the response.``request`` is an object containing keys ``method``, ``url`` and optionally ``body``, ``enable_redirect``, ``force_json_decode``, ``headers``, ``tls_use_system_certs``, ``tls_ca_cert_file``, ``tls_ca_cert_env_variable``, ``tls_client_cert_env_variable``, ``tls_client_key_env_variable`` or ``tls_client_cert_file``, ``tls_client_key_file`` . For example, ``http.send({"method": "get", "url": "http://www.openpolicyagent.org/", "headers": {"X-Foo":"bar", "X-Opa": "rules"}}, output)``. ``output`` is an object containing keys ``status``, ``status_code``, ``body`` and ``raw_body`` which represent the HTTP status, status code, JSON value from the response body and response body as string respectively. Sample output, ``{"status": "200 OK", "status_code": 200, "body": {"hello": "world"}, "raw_body": "{\"hello\": \"world\"}"``}. By default, HTTP redirects are not enabled. To enable, set ``enable_redirect`` to ``true``. Also ``force_json_decode`` is set to ``false`` by default. This means if the HTTP server response does not specify the ``Content-type`` as ``application/json``, the response body will not be JSON decoded ie. output's ``body`` field will be ``null``. To change this behaviour, set ``force_json_decode`` to ``true``.|

#### HTTPs Usage

The following table explains the HTTPs objects

| Object | Definition | Value|
| -------- |-----------|------|
| tls_use_system_certs | Use system certificate pool | true or false
| tls_ca_cert_file | Path to file containing a root certificate in PEM encoded format | double-quoted string
| tls_ca_cert_env_variable | Environment variable containing a root certificate in PEM encoded format | double-quoted string
| tls_client_cert_env_variable | Environment variable containing a client certificate in PEM encoded format | double-quoted string
| tls_client_key_env_variable | Environment variable containing a client key in PEM encoded format | double-quoted string
| tls_client_cert_file | Path to file containing a client certificate in PEM encoded format | double-quoted string
| tls_client_key_file | Path to file containing a key in PEM encoded format | double-quoted string

In order to trigger the use of HTTPs the user must provide one of the following combinations:
| <span class="opa-keep-it-together">``response := http.send(request)``</span> | ``http.send`` executes an HTTP `request` and returns a `response`. |

The `request` object parameter may contain the following fields:

| Field | Required | Type | Description |
| --- | --- | --- | --- |
| `url` | yes | `string` | HTTP URL to specify in the request (e.g., `"https://www.openpolicyagent.org"`). |
| `method` | yes | `string` | HTTP method to specify in request (e.g., `"GET"`, `"POST"`, `"PUT"`, etc.) |
| `body` | no | `any` | HTTP message body to include in request. The value will be serialized to JSON. |
| `raw_body` | no | `string` | HTTP message body to include in request. The value WILL NOT be serialized. Use this for non-JSON messages. |
| `headers` | no | `object` | HTTP headers to include in the request (e.g,. `{"X-Opa": "rules"}`). |
| `enable_redirect` | no | `boolean` | Follow HTTP redirects. Default: `false`. |
| `force_json_decode` | no | `boolean` | Decode the HTTP response message body as JSON even if the `Content-Type` header is missing. Default: `false`. |
| `tls_use_system_certs` | no | `boolean` | Use system certificate pool. |
| `tls_ca_cert_file` | no | `string` | Path to file containing a root certificate in PEM encoded format. |
| `tls_ca_cert_env_variable` | no | `string` | Environment variable containing a root certificate in PEM encoded format. |
| `tls_client_cert_env_variable` | no | `string` | Environment variable containing a client certificate in PEM encoded format. |
| `tls_client_key_env_variable` | no | `string` | Environment variable containing a client key in PEM encoded format. |
| `tls_client_cert_file` | no | `string` | Path to file containing a client certificate in PEM encoded format. |
| `tls_client_key_file` | no | `string` | Path to file containing a key in PEM encoded format. |


To trigger the use of HTTPs the user must provide one of the following combinations:

* ``tls_client_cert_file``, ``tls_client_key_file``
* ``tls_client_cert_env_variable``, ``tls_client_key_env_variable``

The user must also provide a trusted root CA through tls_ca_cert_file or tls_ca_cert_env_variable. Alternatively the user could set tls_use_system_certs to ``true`` and the system certificate pool will be used.

#### HTTPs Examples
The `response` object parameter will contain the following fields:

| Field | Type | Description |
| --- | --- | --- |
| `status` | `string` | HTTP status message (e.g., `"200 OK"`). |
| `status_code` | `number` | HTTP status code (e.g., `200`). |
| `body` | `any` | Any JSON value. If the HTTP response message body was not deserialized from JSON, this field is set to `null`. |
| `raw_body` | `string` | The entire raw HTTP resposne message body represented as a string. |

The table below shows examples of calling `http.send`:

| Examples | Comments |
| Example | Comments |
| -------- |-----------|
| Files containing TLS material | ``http.send({"method": "get", "url": "https://127.0.0.1:65331", "tls_ca_cert_file": "testdata/ca.pem", "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"}, output)``.
|Environment variables containing TLS material | ``http.send({"method": "get", "url": "https://127.0.0.1:65360", "tls_ca_cert_env_variable": "CLIENT_CA_ENV", "tls_client_cert_env_variable": "CLIENT_CERT_ENV", "tls_client_key_env_variable": "CLIENT_KEY_ENV"}, output)``.|
| Accessing Google using System Cert Pool | ``http.send({"method": "get", "url": "https://www.google.com", "tls_use_system_certs": true, "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"}, output)``
| Files containing TLS material | ``http.send({"method": "get", "url": "https://127.0.0.1:65331", "tls_ca_cert_file": "testdata/ca.pem", "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"})`` |
|Environment variables containing TLS material | ``http.send({"method": "get", "url": "https://127.0.0.1:65360", "tls_ca_cert_env_variable": "CLIENT_CA_ENV", "tls_client_cert_env_variable": "CLIENT_CERT_ENV", "tls_client_key_env_variable": "CLIENT_KEY_ENV"})`` |
| Accessing Google using System Cert Pool | ``http.send({"method": "get", "url": "https://www.google.com", "tls_use_system_certs": true, "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"})`` |

### Net
| Built-in | Description |
Expand Down
14 changes: 12 additions & 2 deletions topdown/http.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ var allowedKeyNames = [...]string{
"enable_redirect",
"force_json_decode",
"headers",
"raw_body",
"tls_use_system_certs",
"tls_ca_cert_file",
"tls_ca_cert_env_variable",
Expand Down Expand Up @@ -131,6 +132,7 @@ func executeHTTPRequest(bctx BuiltinContext, obj ast.Object) (ast.Value, error)
var tlsClientCertFile string
var tlsClientKeyFile string
var body *bytes.Buffer
var rawBody *bytes.Buffer
var enableRedirect bool
var forceJSONDecode bool
var tlsUseSystemCerts bool
Expand Down Expand Up @@ -173,6 +175,12 @@ func executeHTTPRequest(bctx BuiltinContext, obj ast.Object) (ast.Value, error)
return nil, err
}
body = bytes.NewBuffer(bodyValBytes)
case "raw_body":
s, ok := obj.Get(val).Value.(ast.String)
if !ok {
return nil, fmt.Errorf("raw_body must be a string")
}
rawBody = bytes.NewBuffer([]byte(s))
case "tls_use_system_certs":
tlsUseSystemCerts, err = strconv.ParseBool(obj.Get(val).String())
if err != nil {
Expand Down Expand Up @@ -211,7 +219,7 @@ func executeHTTPRequest(bctx BuiltinContext, obj ast.Object) (ast.Value, error)
return nil, fmt.Errorf("invalid type for headers key")
}
default:
return nil, fmt.Errorf("Invalid Key %v", key)
return nil, fmt.Errorf("invalid parameter %q", key)
}
}

Expand Down Expand Up @@ -249,7 +257,9 @@ func executeHTTPRequest(bctx BuiltinContext, obj ast.Object) (ast.Value, error)
client.CheckRedirect = nil
}

if body == nil {
if rawBody != nil {
body = rawBody
} else if body == nil {
body = bytes.NewBufferString("")
}

Expand Down
108 changes: 73 additions & 35 deletions topdown/http_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"crypto/tls"
"crypto/x509"
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"net/http"
Expand Down Expand Up @@ -185,59 +186,96 @@ func TestHTTPCustomHeaders(t *testing.T) {
// TestHTTPostRequest adds a new person
func TestHTTPostRequest(t *testing.T) {

// test server
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

var person Person
if r.Body == nil {
http.Error(w, "Please send a request body", 400)
return
}
err := json.NewDecoder(r.Body).Decode(&person)
contentType := r.Header.Get("Content-Type")

bs, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), 400)
return
t.Fatal(err)
}

w.Header().Set("Content-Type", "application/json")
w.Header().Set("Content-Type", contentType)
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(person)
w.Write(bs)
}))

defer ts.Close()

// expected result
expectedResult := map[string]interface{}{
"status": "200 OK",
"status_code": http.StatusOK,
"body": map[string]string{"id": "2", "firstname": "Joe"},
"raw_body": "{\"id\":\"2\",\"firstname\":\"Joe\"}\n",
}

resultObj, err := ast.InterfaceToValue(expectedResult)
if err != nil {
panic(err)
}

// create a new person object
person2 := Person{ID: "2", Firstname: "Joe"}
b := new(bytes.Buffer)
json.NewEncoder(b).Encode(person2)

// run the test
tests := []struct {
note string
rules []string
params string
expected interface{}
}{
{"http.send", []string{fmt.Sprintf(
`p = x { http.send({"method": "post", "url": "%s", "body": %s}, x) }`, ts.URL, b)}, resultObj.String()},

{
note: "basic",
params: `{
"method": "post",
"headers": {"Content-Type": "application/json"},
"body": {"id": "2", "firstname": "Joe"}
}`,
expected: `{
"status": "200 OK",
"status_code": 200,
"body": {"id": "2", "firstname": "Joe"},
"raw_body": "{\"firstname\":\"Joe\",\"id\":\"2\"}"
}`,
},
{
note: "raw_body",
params: `{
"method": "post",
"headers": {"Content-Type": "application/x-www-form-encoded"},
"raw_body": "username=foobar&password=baz"
}`,
expected: `{
"status": "200 OK",
"status_code": 200,
"body": null,
"raw_body": "username=foobar&password=baz"
}`,
},
{
note: "raw_body overrides body",
params: `{
"method": "post",
"headers": {"Content-Type": "application/x-www-form-encoded"},
"body": {"foo": 1},
"raw_body": "username=foobar&password=baz"
}`,
expected: `{
"status": "200 OK",
"status_code": 200,
"body": null,
"raw_body": "username=foobar&password=baz"
}`,
},
{
note: "raw_body bad type",
params: `{
"method": "post",
"headers": {"Content-Type": "application/x-www-form-encoded"},
"raw_body": {"bar": "bar"}
}`,
expected: errors.New("raw_body must be a string"),
},
}

data := loadSmallTestData()
data := map[string]interface{}{}

for _, tc := range tests {
runTopDownTestCase(t, data, tc.note, tc.rules, tc.expected)

// Automatically set the URL because it's generated when the test server
// is started. If needed, the test cases can override in the future.
term := ast.MustParseTerm(tc.params)
term.Value.(ast.Object).Insert(ast.StringTerm("url"), ast.StringTerm(ts.URL))

rules := []string{
fmt.Sprintf(`p = x { http.send(%s, x) }`, term),
}

runTopDownTestCase(t, data, tc.note, rules, tc.expected)
}
}

Expand Down
2 changes: 1 addition & 1 deletion topdown/topdown_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2979,7 +2979,7 @@ func assertTopDownWithPath(t *testing.T, compiler *ast.Compiler, store storage.S
}

if util.Compare(expected, result) != 0 {
t.Fatalf("Unexpected result:\nGot: %v\nExp:\n%v", result, expected)
t.Fatalf("Unexpected result:\nGot: %+v\nExp:\n%+v", result, expected)
}

// If the test case involved the input document, re-run it with partial
Expand Down

0 comments on commit cdcebd8

Please sign in to comment.