Specifying the port # in a URL #2568

paulmckeown · 2020-07-22T11:19:04Z

When using a URL of the form

url: http://10.246.22.181:9020/opabckt1

on starting OPA the attempt to read the bundles returns signature mismatch errors.

If I change the url to not specify the port number via an IP load balancer to for example

url: http://my_load_balancer.com/opabckt1

OPA is able to read the bundles with no issues.

I would expect to be able to specify the port number by appending the port as in the above example.

OPA version: v0.22.0 which I believe is the latest

I'm running OPA on

Linux CentOS 7
Kernel: Linux 3.10.0-1127.13.1.el7.x86_64

config.yaml

services:
s3:
url: http://10.246.22.181:9020/opabckt1

credentials:
  s3_signing:
    environment_credentials:
      aws_access_key_id: ${AWS_ACCESS_KEY_ID}
      aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}
      aws_region: ${AWS_REGION}

bundles:
policy:
service: s3
resource: policy.tar.gz
polling:
min_delay_seconds: 10
max_delay_seconds: 30

data:
service: s3
resource: data.tar.gz
polling:
min_delay_seconds: 10
max_delay_seconds: 30

The S3 target is a Dell EMC ECS S3 object store revision 3.5 and 3.4 (both have been tested)

Could I get confirmation that this is the correct way to specify the port # on an S3 type url and if specifying the port number like this is causing the signature mismatch I'm seeing?

Thanks Paul

The text was updated successfully, but these errors were encountered:

paulmckeown · 2020-07-22T13:25:40Z

To add to this I just quickly installed minio (latest version) and I get the same issue with it.

patrick-east · 2020-07-22T16:53:07Z

Thanks for filing the issue!

Any chance you could post the steps you took to reproduce with minio? (just to save us some time to figure out how to configure it 😄 )

Could I get confirmation that this is the correct way to specify the port # on an S3 type url and if specifying the port number like this is causing the signature mismatch I'm seeing?

That does look like the correct formatting for the url parameter of the service. I suspect that most users are using AWS S3 so there isn't typically a port in the url (there are other issues however around proxies #2042 ).

paulmckeown · 2020-07-22T17:35:44Z

Sure it's pretty easy to install and configure (I hadn't used it before this)

I just used windows but you can install it on linux if you want

Download it here https://min.io/download#/windows it's an executable

set the following environment variables

MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin

In windows I created a folder c:\minio which is for it's object store and config files etc

then execute it as

minio server c:\minio

If will start up and you can connect to it with a browser on port 127.0.0.1:9000 or the servers ip address:9000 from other workstations.

the admin account by default is minioadmin with minioadmin as the password. You can use this account for s3 data as well. use the minioadmin as the s3 secret.

There is a client mc.exe you can download and use but you don't have to.

The browser will let you create a bucket and you can use s3browser or other s3 util to upload your bundles.

Then just set your url to url: http://minio_ip:9000/mybucket and off you go

There's probably a lot more to setting minio up but this was enough for me to test it.

I guess all your current users are just using a cloud server so never need to specify a port #. This is an on_prem installation and the default ports can't be used to map them to what is required

I think I've got the steps down for you okay, sorry if any mistakes crept in

patrick-east · 2020-07-22T20:27:21Z

Perfect 👍 Thanks for the instructions.

The AWS v4 signing feature for bundle requests would automatically add a `Host` header by using the URL hostname. This would break for URLs that specified a port number (eg: https://127.0.0.1:9000/). That causes issues with sending a valid request. This commit changes to use the `URL.Host` which will have the full `host:port` string, when a port was specified. Fixes: open-policy-agent#2568 Signed-off-by: Patrick East <east.patrick@gmail.com>

patrick-east · 2020-07-22T21:52:54Z

I think I found the issue, or at least got it to work with minio. The Host header wasn't including the port.

@paulmckeown would you be able to give the code in #2572 a try with your Dell EMC ECS S3 object store to ensure it corrects the original issue too?

The AWS v4 signing feature for bundle requests would automatically add a `Host` header by using the URL hostname. This would break for URLs that specified a port number (eg: https://127.0.0.1:9000/). That causes issues with sending a valid request. This commit changes to use the `URL.Host` which will have the full `host:port` string, when a port was specified. Fixes: #2568 Signed-off-by: Patrick East <east.patrick@gmail.com>

patrick-east · 2020-07-22T22:10:08Z

Update, we went ahead with merging this. Turns out the issue reproduces with "real" s3 if you put the :443 on the configured URL, and it is corrected by #2572.

It would still be good to test it out and ensure we're good to have this issue closed. It's easier now though, try out the latest edge release ( https://www.openpolicyagent.org/docs/edge/#1-download-opa or edge version tag on docker image).

paulmckeown · 2020-07-23T09:40:06Z

Patrick I tried the latest release via the link above but it still fails for minio and for ECS is it the correct link?
Paul
Paul.mckeown@dell.com

paulmckeown · 2020-07-23T10:36:38Z

Also if you wish to test against Dell EMC ECS we have an online ECS which our partners and customers can use for functional testing. You can sign up for credentials at

https://portal.ecstestdrive.com/

It is behind a Load Balancer but you can add the :443 port to the URL to test your fix.

patrick-east · 2020-07-23T17:37:35Z

Thanks! I signed up on the test drive site (waiting for email confirmation to get access).

For minio I am seeing it work, my steps are:

Start minio:

{15:31} ~/p/g/s/g/o/opa:issue/2568 ✓ ❯ minio server /tmp/minio-data
Endpoint:  http://10.1.0.142:9000  http://10.1.12.8:9000  http://127.0.0.1:9000
AccessKey: minioadmin
SecretKey: minioadmin

Browser Access:
   http://10.1.0.142:9000  http://10.1.12.8:9000  http://127.0.0.1:9000

Command-line Access: https://docs.min.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://10.1.0.142:9000 minioadmin minioadmin

Object API (Amazon S3 compatible):
   Go:         https://docs.min.io/docs/golang-client-quickstart-guide
   Java:       https://docs.min.io/docs/java-client-quickstart-guide
   Python:     https://docs.min.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide
Detected default credentials 'minioadmin:minioadmin', please change the credentials immediately using 'MINIO_ACCESS_KEY' and 'MINIO_SECRET_KEY'

In another terminal (I already had the mc config setup from playing with it yesterday):

{10:26} ~/p/g/s/g/o/opa:issue/2568 ✓ ❯ mc config host list myminio
myminio
  URL       : http://10.1.0.142:9000
  AccessKey : minioadmin
  SecretKey : minioadmin
  API       : s3v4
  Lookup    : auto

Which confirms the host is setup for s3v4 signing

{10:27} ~/p/g/s/g/o/opa:issue/2568 ✓ ❯ cat /tmp/config.yaml
services:
    s3:
        url: http://10.1.0.142:9000/
        credentials:
            s3_signing:
                environment_credentials: {}

bundles:
    policy:
        service: s3
        resource: opa/bundle.tar.gz
        polling:
            min_delay_seconds: 5
            max_delay_seconds: 5

I've got OPA configured to use that service and I tell it to look for environment credentials.. just need to upload a test bundle..

{10:27} ~/p/g/s/g/o/opa:issue/2568 ✓ ❯ mc mb myminio/opa
Bucket created successfully `myminio/opa`.
{10:27} ~/p/g/s/g/o/opa:issue/2568 ✓ ❯ mc cp /tmp/bundle.tar.gz myminio/opa/
/tmp/bundle.tar.gz:                       210 B / 210 B  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  10.01 KiB/s 0s
{10:27} ~/p/g/s/g/o/opa:issue/2568 ✓ ❯ mc find myminio/opa
myminio/opa/bundle.tar.gz

And now getting the edge release (in this case for osx, but the url/steps should be similar for other platforms)

{10:28} ~/p/g/s/g/o/opa:issue/2568 ✓ ❯ curl -L -o opa https://openpolicyagent.org/downloads/edge/opa_darwin_amd64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    79  100    79    0     0    443      0 --:--:-- --:--:-- --:--:--   441
100 20.3M  100 20.3M    0     0   838k      0  0:00:24  0:00:24 --:--:--  880k
{10:29} ~/p/g/s/g/o/opa:issue/2568 ✓ ❯ chmod +x ./opa

Last step is to configure the environment variables that will be read for the credentials and start opa:

{10:28} ~/p/g/s/g/o/opa:issue/2568 ✓ ❯ export AWS_REGION=us-east-1
{10:28} ~/p/g/s/g/o/opa:issue/2568 ✓ ❯ export AWS_ACCESS_KEY_ID=minioadmin
{10:28} ~/p/g/s/g/o/opa:issue/2568 ✓ ❯ export AWS_SECRET_ACCESS_KEY=minioadmin
{10:29} ~/p/g/s/g/o/opa:issue/2568 ✓ ❯ ./opa run --server --config-file /tmp/config.yaml --log-level debug --log-format text
[INFO] Initializing server.
  insecure_addr = ""
  addrs = |
      [
        ":8181"
      ]
  diagnostic-addrs = []

[INFO] Starting bundle downloader.
  plugin = "bundle"
  name = "policy"

[DEBUG] Download starting.

[DEBUG] Signing request with AWS credentials.

[DEBUG] Sending request.
  method = "GET"
  url = "http://10.1.0.142:9000/opa/bundle.tar.gz"
  headers = |
      {
        "Authorization": [
          "AWS4-HMAC-SHA256 Credential=minioadmin/20200723/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=9c950cf8ec337379bc3f66a5f99a94e614e6a1f80414d730a1cccc25fe6d4ad4"
        ],
        "Host": [
          "10.1.0.142:9000"
        ],
        "User-Agent": [
          "Open Policy Agent/0.23.0-dev (darwin, amd64)"
        ],
        "X-Amz-Content-Sha256": [
          "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
        ],
        "X-Amz-Date": [
          "20200723T173029Z"
        ]
      }

[DEBUG] Sending request.
  method = "POST"
  url = "https://telemetry.openpolicyagent.org/v1/version"
  headers = |
      {
        "Content-Type": [
          "application/json"
        ],
        "User-Agent": [
          "Open Policy Agent/0.23.0-dev (darwin, amd64)"
        ]
      }

[DEBUG] Received response.
  method = "GET"
  url = "http://10.1.0.142:9000/opa/bundle.tar.gz"
  status = "200 OK"
  headers = |
      {
        "Accept-Ranges": [
          "bytes"
        ],
        "Content-Length": [
          "210"
        ],
        "Content-Security-Policy": [
          "block-all-mixed-content"
        ],
        "Content-Type": [
          "application/gzip"
        ],
        "Date": [
          "Thu, 23 Jul 2020 17:30:29 GMT"
        ],
        "Etag": [
          "\"d27d134ec751d77be7dbab9aff37d470\""
        ],
        "Last-Modified": [
          "Thu, 23 Jul 2020 17:27:53 GMT"
        ],
        "Server": [
          "MinIO/RELEASE.2020-07-22T00-26-33Z"
        ],
        "Vary": [
          "Origin"
        ],
        "X-Amz-Request-Id": [
          "162471ECAFFDBAE0"
        ],
        "X-Xss-Protection": [
          "1; mode=block"
        ]
      }

[DEBUG] Download in progress.

[DEBUG] Bundle activation in progress. Opening storage transaction.
  plugin = "bundle"
  name = "policy"

[DEBUG] Opened storage transaction (3).
  plugin = "bundle"
  name = "policy"

[DEBUG] Closing storage transaction (3).
  name = "policy"
  plugin = "bundle"

[INFO] Bundle downloaded and activated successfully. Etag updated to "d27d134ec751d77be7dbab9aff37d470".
  plugin = "bundle"
  name = "policy"

[DEBUG] Waiting 5s before next download/retry.

^C[INFO] Shutting down...

[INFO] Server shutdown.

[INFO] Stopping bundle downloader.
  plugin = "bundle"
  name = "policy"

In the log I'm seeing that it was able to successfully pull the bundle.

For your environment please confirm in the debug output if the Host header is including the port information, that should confirm whether it's the correct version of OPA running.

paulmckeown · 2020-07-23T17:40:55Z

Ok I'll try again.

I approved your ECS portal account so you should have your credentials now

paulmckeown · 2020-07-23T18:12:59Z

It works I must've fat fingered something this morning

Thanks for quick fix!

patrick-east · 2020-07-23T18:15:29Z

Trying it out with the ECS Test Drive:

{11:01} ~/p/g/s/g/o/opa:issue/2568 ◈ ❯ mc config host add ecstestdrive https://object.ecstestdrive.com --api "s3v4"
Enter Access Key: ************************@ecstestdrive.emc.com
Enter Secret Key:
Added `ecstestdrive` successfully.
{11:02} ~/p/g/s/g/o/opa:issue/2568 ◈ ❯ mc ls ecstestdrive/
{11:03} ~/p/g/s/g/o/opa:issue/2568 ◈ ❯ mc mb ecstestdrive/opa-2568
Bucket created successfully `ecstestdrive/opa-2568`.
{11:04} ~/p/g/s/g/o/opa:issue/2568 ◈ ❯ mc cp /tmp/bundle.tar.gz ecstestdrive/opa-2568/
/tmp/bundle.tar.gz:                       210 B / 210 B  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  98 B/s 2s

Quick sanity check in Cyberduck (wasn't sure if the mc tool would work, turns out to be pretty helpful for this stuff):

Updated opa config (using :443 on the url):

{11:07} ~/p/g/s/g/o/opa:issue/2568 ◈ ❯ vi /tmp/config.yaml
{11:10} ~/p/g/s/g/o/opa:issue/2568 ◈ ❯ cat /tmp/config.yaml
services:
    s3:
        url: https://object.ecstestdrive.com:443/
        credentials:
            s3_signing:
                environment_credentials: {}

bundles:
    policy:
        service: s3
        resource: opa-2568/bundle.tar.gz
        polling:
            min_delay_seconds: 5
            max_delay_seconds: 5

Updated credentials, the region thing gets defaulted by mc to us-east-1, not 100% sure how that works for ECS (or minio for that matter) if you use another tool. It does get used in the signature so it needs to match up with the bucket config.

{11:10} ~/p/g/s/g/o/opa:issue/2568 ◈ ❯ export AWS_REGION=us-east-1
{11:10} ~/p/g/s/g/o/opa:issue/2568 ◈ ❯ export AWS_ACCESS_KEY_ID=********************@ecstestdrive.emc.com
{11:11} ~/p/g/s/g/o/opa:issue/2568 ◈ ❯ export AWS_SECRET_ACCESS_KEY=***************************************

And running with the updated config:

{11:11} ~/p/g/s/g/o/opa:issue/2568 ◈ ❯ ./opa run --server --config-file /tmp/config.yaml --log-level debug --log-format text
[INFO] Initializing server.
  diagnostic-addrs = []
  insecure_addr = ""
  addrs = |
      [
        ":8181"
      ]

[INFO] Starting bundle downloader.
  name = "policy"
  plugin = "bundle"

[DEBUG] Download starting.

[DEBUG] Signing request with AWS credentials.

[DEBUG] Sending request.
  method = "GET"
  url = "https://object.ecstestdrive.com:443/opa-2568/bundle.tar.gz"
  headers = |
      {
        "Authorization": [
          "AWS4-HMAC-SHA256 Credential=**************************@ecstestdrive.emc.com/20200723/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=e8d3d05fbea490a4b79923e4902fc15da432e58f37ce21b108b42111b69e7040"
        ],
        "Host": [
          "object.ecstestdrive.com:443"
        ],
        "User-Agent": [
          "Open Policy Agent/0.23.0-dev (darwin, amd64)"
        ],
        "X-Amz-Content-Sha256": [
          "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
        ],
        "X-Amz-Date": [
          "20200723T181128Z"
        ]
      }

[DEBUG] Sending request.
  method = "POST"
  url = "https://telemetry.openpolicyagent.org/v1/version"
  headers = |
      {
        "Content-Type": [
          "application/json"
        ],
        "User-Agent": [
          "Open Policy Agent/0.23.0-dev (darwin, amd64)"
        ]
      }

[DEBUG] Received response.
  method = "GET"
  url = "https://object.ecstestdrive.com:443/opa-2568/bundle.tar.gz"
  status = "200 OK"
  headers = |
      {
        "Content-Length": [
          "210"
        ],
        "Content-Type": [
          "application/gzip"
        ],
        "Date": [
          "Thu, 23 Jul 2020 18:08:36 GMT"
        ],
        "Etag": [
          "\"d27d134ec751d77be7dbab9aff37d470\""
        ],
        "Last-Modified": [
          "Thu, 23 Jul 2020 18:02:05 GMT"
        ],
        "Server": [
          "ViPR/1.0"
        ],
        "X-Amz-Id-2": [
          "d1657217f796e4965adfa2f3244865c233bd662f7b83a08765e485f77d7fe22f"
        ],
        "X-Amz-Request-Id": [
          "0c07c833:1735dbe5f25:9755:329"
        ],
        "X-Emc-Mtime": [
          "1595527325667"
        ]
      }

[DEBUG] Download in progress.

[DEBUG] Bundle activation in progress. Opening storage transaction.
  plugin = "bundle"
  name = "policy"

[DEBUG] Opened storage transaction (3).
  plugin = "bundle"
  name = "policy"

[DEBUG] Closing storage transaction (3).
  plugin = "bundle"
  name = "policy"

[INFO] Bundle downloaded and activated successfully. Etag updated to "d27d134ec751d77be7dbab9aff37d470".
  plugin = "bundle"
  name = "policy"

[DEBUG] Waiting 5s before next download/retry.

^C[INFO] Shutting down...

[INFO] Server shutdown.

[INFO] Stopping bundle downloader.
  plugin = "bundle"
  name = "policy"

Looks like everything is working ok. This is with OPA edge as of commit dd68302:

{11:14} ✘ ~/p/g/s/g/o/opa:issue/2568 ◈ ❯ ./opa version
Version: 0.23.0-dev
Build Commit: dd68302
Build Timestamp: 2020-07-22T22:19:48Z
Build Hostname: bdc8ce71c3cf

paulmckeown · 2020-07-23T18:59:52Z

great it worked for you as well so that confirms it

BTW - ECS doesn't care what you put for region it ignores it.

The credential you have will stay active for ever and yo have about 1TB of storage you can use so feel free to keep using it as another QA target.

If you run into issues using it email me on paul.mckeown@dell.com

Paul

patrick-east added the bug label Jul 22, 2020

patrick-east self-assigned this Jul 22, 2020

patrick-east added the investigating Issues being actively investigated label Jul 22, 2020

patrick-east mentioned this issue Jul 22, 2020

plugins/rest/aws: Include port in Host header #2572

Merged

patrick-east closed this as completed in #2572 Jul 22, 2020

patrick-east reopened this Jul 23, 2020

tsandall closed this as completed Jul 23, 2020

ashutosh-narkar added this to Open Policy Agent Aug 5, 2024

ashutosh-narkar moved this to Done in Open Policy Agent Aug 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Specifying the port # in a URL #2568

Specifying the port # in a URL #2568

paulmckeown commented Jul 22, 2020 •

edited

Loading

paulmckeown commented Jul 22, 2020

patrick-east commented Jul 22, 2020

paulmckeown commented Jul 22, 2020

patrick-east commented Jul 22, 2020

patrick-east commented Jul 22, 2020

patrick-east commented Jul 22, 2020 •

edited

Loading

paulmckeown commented Jul 23, 2020

paulmckeown commented Jul 23, 2020

patrick-east commented Jul 23, 2020

paulmckeown commented Jul 23, 2020

paulmckeown commented Jul 23, 2020

patrick-east commented Jul 23, 2020

paulmckeown commented Jul 23, 2020

Specifying the port # in a URL #2568

Specifying the port # in a URL #2568

Comments

paulmckeown commented Jul 22, 2020 • edited Loading

paulmckeown commented Jul 22, 2020

patrick-east commented Jul 22, 2020

paulmckeown commented Jul 22, 2020

patrick-east commented Jul 22, 2020

patrick-east commented Jul 22, 2020

patrick-east commented Jul 22, 2020 • edited Loading

paulmckeown commented Jul 23, 2020

paulmckeown commented Jul 23, 2020

patrick-east commented Jul 23, 2020

paulmckeown commented Jul 23, 2020

paulmckeown commented Jul 23, 2020

patrick-east commented Jul 23, 2020

paulmckeown commented Jul 23, 2020

paulmckeown commented Jul 22, 2020 •

edited

Loading

patrick-east commented Jul 22, 2020 •

edited

Loading