aktualizr fails to download reposerver files backed by S3

[doanac]

I'm going to try and dig into this, but before I get too far I thought I'd describe the issue in case someone happens to already know the root issue. I have a tufreposerver deployed that's backed by an S3 bucked. When I call PackageManagerInterface::fetchTarget it gets the 302 redirect to a signed URL, but then the download fails. It looks like it may be something with the HttpClient sending the gateway certs to my S3 bucked (in my case its a google storage acting as s3):

   info: Looking for DockerApps to fetch
   info: Fetching shellhttpd -> Target(shellhttpd.dockerapp-98 ecu_identifiers: () length:312 hashes: (Hash: 7A26E16871423505EB0B92272680FCE4EFFBA1316259F91312CE7F27D42F8E2E, ))
*   Trying 35.222.49.182...
* TCP_NODELAY set
* Connected to ota-lite.foundries.io (35.222.49.182) port 8443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /tmp/aktualizr-7dce-ebe6-f48a-0673/011d-e559-tls-ca
  CApath: none
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=ota-gateway
*  start date: Jun 14 03:35:51 2019 GMT
*  expire date: Jun 11 03:35:51 2029 GMT
*  subjectAltName: host "ota-lite.foundries.io" matched cert's "ota-lite.foundries.io"
*  issuer: CN=ota-server-CA
*  SSL certificate verify ok.
> GET /repo/targets/shellhttpd.dockerapp-98 HTTP/1.1
Host: ota-lite.foundries.io:8443
User-Agent: Aktualizr/1.0+gitAUTOINC+4397bdcfed
Content-Type: application/json
Accept: */*
x-ats-ostreehash: 4b16e43adbfa25eab74eaf5822a99209c085e7d01665a98af951a81b2e9be7a7

< HTTP/1.1 302 Found
< Server: openresty/1.15.8.1
< Date: Tue, 30 Jul 2019 04:23:27 GMT
< Content-Length: 0
< Connection: keep-alive
< x-ats-version: reposerver/0.6.0-19-ge074093
< x-ats-tuf-repo-id: de7b16d5-2bdb-4b7a-8932-498805997f09
< Location: https://andy-treehub-test.storage.googleapis.com/13507a5695ad52638769c65040efa0eb5f7e5c8cc058aa76ae840f68c9880706/b7/fdf692ebb39acf8af819742aa60fba520ea5433e71600a27779e22d1565fbb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20190730T042327Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=GOOGIQOQXOYO4CHMDY4EEW5M%2F20190730%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=911828466049e795752af7dbfb27789b4821064a867347afee09ba5ae5db526a
< 
* Connection #0 to host ota-lite.foundries.io left intact
* Issue another request to this URL: 'https://andy-treehub-test.storage.googleapis.com/13507a5695ad52638769c65040efa0eb5f7e5c8cc058aa76ae840f68c9880706/b7/fdf692ebb39acf8af819742aa60fba520ea5433e71600a27779e22d1565fbb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20190730T042327Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=GOOGIQOQXOYO4CHMDY4EEW5M%2F20190730%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=911828466049e795752af7dbfb27789b4821064a867347afee09ba5ae5db526a'
*   Trying 172.217.1.144...
* TCP_NODELAY set
* Connected to andy-treehub-test.storage.googleapis.com (172.217.1.144) port 443 (#1)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /tmp/aktualizr-7dce-ebe6-f48a-0673/011d-e559-tls-ca
  CApath: none
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 1
* Closing connection 0

This is a bit strange to me because aktualizr has no problems talking to TreeHub which is backed by the exact same S3 bucket. I'll do some poking around tonight, but was curious if anybody had any clues?

[doanac]

Wild issue. I think I've sorted things out. I've been toying with a new tool, aktualizr-get which helped out on this. When I ran the tool inside the aktualizr container things were working. When I ran them on my rpi3 OE build it failed. I then isolated it to just running the command with the alpine version of curl versus the debian version. I'm guessing debian is carrying around some distro patch that ensures curl is always looking at /etc/ssl/certs because when I added this hack:

doanac@2f7f2c0

The bug goes away. I'm not sure if that patch is the way you want this. Feel free to fixing another way or letting me know how you'd like me to proceed.

[pattivacek]

Ah, interesting. I'm not completely sure what's going on in libaktualizr, but the garage tools do set CURLOPT_CAPATH and maybe that is the right thing to do. It should probably be configurable, though. @lbonn @eu-smirnov any concerns with that?

[doanac]

@rsalveti - has been looking at this a bit. He's found that OE and Alpine both build curl differently than Debian. Quoting him:

one difference I noticed from debian/OE
cd debian/build && dh_auto_configure ${CONFIGURE_ARGS}
--with-ca-path=/etc/ssl/certs
--with-ca-bundle=/etc/ssl/certs/ca-certificates.crt
this is on debian
OE only sets ca-bundle
alpine doesn't set any of those with-ca options

he's keen on fixing curl in OE right now. But we may want to fix things here also just to prevent someone from debugging the same thing I just did

[ricardosalveti]

Right, the issue here is that when you set a ca-certificate via cacert that ends up replacing the default ca-bundle, so curl is unable to find additional certificates if needed (e.g. after redirects). Setting curl to also have ca-path by default would fix the issue, but another fix would be to also set it here (which is probably better as it would work on any system).

[ricardosalveti]

So my thinking is that we should actually fix on both places (OE to avoid users to face the same issue and aktualizr to make sure it works correctly across multiple systems).

[pattivacek]

So my thinking is that we should actually fix on both places (OE to avoid users to face the same issue and aktualizr to make sure it works correctly across multiple systems).

That sounds like a good plan. Does it sound like a reasonable plan to have a parameter in our configuration that is a path to the CA certs directory? I don't like having it hardcoded, although it's fine for /etc/ssl/certs to be the default.

[doanac]

@patrickvacek - I think that makes sense. I just got back from vacation and will get some time to finish this up early next week.

[doanac]

@patrickvacek - just looked at doing your suggestion a little more closely. I think the problem with making it a config variable is that its going to require all call sites of HttpClient to now pass in a reference to the Config object. I'm thinking it might be easier if we make this a CMake option like -DSSL_CERT_PATH=<value or /etc/ssl/certs>

make sense?

[pattivacek]

If the cert path was passed to the HttpClient as an optional string, I think that would be fine. I don't particularly like expanding the CMake options if we can help it. That makes things harder to test and less flexible in general.

[eu-siemann]

This change basically means that in our communication with device gateway we will trust any certificate signed by the CA in the OS bundle. IMO, this can open us to a range of attacks if one of the CAs gets compromised.
@zabbal What's your view on that?

[zabbal]

This should not pose any immediate risk unless we have implemented Uptane incorrectly: the framework security guarantees are independent of transport channel security - even if we work over completely open channel.

Having said that I do think that we should double-check that this parameter is clearly documented: some integrators might want to explicitly set CURLOPT_CAPATH to something they control (or even to /nonexistent) without interferring with system-wide certificate store.

There's been reports of compromised public CA (those you normally expect to be in /etc/ssl/certs) by individuals [1] and organizations [2] as well as government actors [3] actively seeking to get into the default CA list [3].

On the one hand it's reasonable to expect that on hardened embedded system the /etc/ssl/certs is tightly controlled, on the other hand it's one of those details which is trivial to overlook.

After looking at https://docs.ota.here.com/ota-client/latest/aktualizr-config-options.html I can't find anything about CURLOPT_CAPATH - am I looking at the wrong place?

Exec summary: having this option is good (it's not up to us to decide client's security policy on which CAs to trust) but it has to be properly documented to make sure that client makes an informed decision instead of whatever-works-default-what's-CA-anyway kind of thing.

P. S> And doh! on me for overlooking the related commit.

[1] https://blog.erratasec.com/2011/03/comodo-hacker-releases-his-manifesto.html
[2] https://blog.isc2.org/isc2_blog/2012/04/test.html
[2] https://thehackernews.com/2019/08/kazakhstan-root-certificate.html

[eu-siemann]

This should not pose any immediate risk unless we have implemented Uptane incorrectly: the framework security guarantees are independent of transport channel security - even if we work over completely open channel.

Like receiving the initial root.json over this channel?

[zabbal]

Yepp, we're completely screwed in this case. Though the credentials.zip is almost inevitably retrieved via browser which uses the same CA list so if attacker can subvert one of the CAs than he can already substitute URL from which initial root.json will be retrieved. Hence I'm not sure how to assess this risk. Any ideas?

Another scenario to consider is when attacker managed to inject redirection to the URL he controls (and for which he can obtain legitimate cert from one of the default CAs) into the endpoint serving root.json. In this case not using the default CA list would certainly help but the likelyhood of compromise that allows redirect but does not allow to replace the root.json is an open question.

Overall I think it boils down to what default we set for CURLOPT_CAPATH: /etc/ssl/certs or /nonexistent - whether we think that risk from latter scenario outweights the confusion as described in this ticket.

Thoughts?

Regardless of that choice I think we should update documentation to reflect this.

[eu-siemann]

It might be also useful to distinguish between metadata and image fetching. We can set the default CAPATH to /nonexistent for the former and to /etc/ssl/certs for the latter. This should harden security in the aforementioned scenarios without breaking custom-url use cases. I, of course, agree that we still should make this configurable and reflect in documentation.