Intermediate certificates required to be in CA store

[ygale]

Given this certificate path:

GeoTrust Global CA (issued 2002-05-21) > GeoTrust SSL CA - G3 (issued 2013-11-06) > user certificate

On Windows, browsers (at least Chrome and IE) can access the site without error, but our Haskell program that tries to access the site programmatically fails with a "CA unknown" error. We can resolve the problem by manually installing the intermediate certificate into the Windows system CA store. But that causes serious difficulties when deploying our software inside corporate networks.

Why does tls require also the intermediate certificate to be in the CA store, when standard browsers do not? How would you suggest working around this problem?

[vincenthz]

It doesn't though: the certificate chains is traversed and at each step, the certificate is checked against the trusted store.

[ygale]

Oh, hmm. So maybe we are not getting the intermediate cert from the server for some reason, whereas those browsers are retrieving it. Could that be it? Is this part of TLS negotiation?

FYI we are using http-client-tls via http-conduit.

[vincenthz]

No, usually the certificates that the server is going to send doesn't depend on the clients.

chrome has it own CA store, AFAIK, so it's not surprising. IE on the other hand should use the same source (registry). it would probably be a good idea to have an utility to test which certificates are available in the store.

[crossleydominic]

This appears to be an older issue but I'd thought I'd add some more (hopefully useful) information.
We've recently encountered this problem on two occasions when interacting with two external HTTP API service providers. In both cases the problem appeared to be caused by a misconfiguration by the vendor of their SSL certificates. During the SSL negotiation the intermediate certificates were not presented to our client (nor were they in our local store) which caused the certificate chain validation to fail.

I'm not intimately familiar with the TLS specification so I can't say for sure whether the vendor is at fault and whether or not it is their responsibility to correctly return the intermediate certificates during the SSL negotiation.

I believe reason that some web browsers are able to correctly validate the certification chain is due to the "Authority Information Access (1.3.6.1.5.5.7.1.1)" TLS extension being present on the leaf certificate. This extension provides information allowing for missing intermediate certificates to be obtained during the chain validation process, thereby allowing the validation to succeed.

I lieu of this extension being implemented in tls we came up with two workarounds:

1. Provide a custom validation function that augments the existing validation. BEWARE, how you validate the intermediate certificate is up to you, you could potentially leave yourself vulnerable to a man-in-the-middle attack. Example code (apologies for the formatting)

{-# LANGUAGE OverloadedStrings #-}

module Main where

import           Control.Lens             ((&), (.~))
import qualified Control.Monad.IO.Class   as IO
import qualified Data.ASN1.Types          as ASN1
import qualified Data.ByteString          as BS
import qualified Data.Default.Class       as Def
import qualified Data.Maybe as Mb
import qualified Data.X509                as X509             -- from 'x509'
import qualified Data.X509.Validation     as X509.V           -- from 'x509-validation'
import qualified Network.Connection       as Net.Conn         -- from 'connection'
import qualified Network.HTTP.Client.TLS  as HTTP.Client.TLS  -- from 'http-client-tls'
import qualified Network.TLS              as Net.TLS          -- from 'tls'
import qualified Network.TLS.Extra        as Net.TLS.Ex       -- from 'tls'
import qualified System.X509              as X509             -- from 'x509-system'

import qualified Network.Wreq as W

unsafeCertificateValidation :: IO.MonadIO m => m W.Options
unsafeCertificateValidation
  = do
      systemCertStore <- IO.liftIO X509.getSystemCertificateStore

      let tlsClientParams
            = (Net.TLS.defaultParamsClient "" BS.empty)
                { Net.TLS.clientShared
                    = Def.def { Net.TLS.sharedCAStore = systemCertStore
                              , Net.TLS.sharedValidationCache = Def.def
                              }

                , Net.TLS.clientHooks
                    = Def.def { Net.TLS.onServerCertificate
                                  = \cs vc sid cc -> do
                                      if isInvalidCertificateChainVendor cc
                                        then pure []
                                        else X509.V.validateDefault cs vc sid cc
                              }
                }

          tlsSettings = Net.Conn.TLSSettings tlsClientParams

          managerSettings  = HTTP.Client.TLS.mkManagerSettings tlsSettings Nothing

          options = W.defaults & W.manager .~ Left managerSettings
                               & W.checkStatus .~ Just (\_ _ _ -> Nothing)

      pure options

isInvalidCertificateChainVendor :: X509.CertificateChain -> Bool
isInvalidCertificateChainVendor (X509.CertificateChain signedExactList)
  = case signedExactList of
      [signedExact] ->
        let (X509.Signed c _ _) = X509.getSigned signedExact
            serial              = X509.certSerial c
            issuerDN            = Mb.fromMaybe "<MISSING>"
                                    (ASN1.asn1CharacterToString =<<
                                      (X509.getDnElement X509.DnCommonName
                                        (X509.certIssuerDN c)))

        in  if -- The integer representation of the vendor certificate serial number
               serial == 34447982775586662229568477121756425455 && 

               -- The distinguished name of the vendors issuer
               issuerDN == "Vendor intermediate distinguished name here"

               then -- Do custom validation here
               else False

      _ -> False

main :: IO ()
 main
   = do
      putStrLn "Getting..."
      opts <- unsafeCertificateValidation
      r <- W.getWith opts "<vendor url here>"
      putStrLn "Got..."
      print r

2. Provide a custom certificate store which contains the missing certificates. These missing certificates can be bundled with your apps binary or, as in the example below, compiled in for a low-fi solution. Example code:

{-# LANGUAGE NamedFieldPuns    #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE ScopedTypeVariables #-}

module Main where

import           Control.Lens             ((&), (.~))
import           Data.Monoid              ((<>))
import qualified Control.Monad.IO.Class   as IO
import qualified Data.ByteString          as BS
import qualified Data.ByteString.Base64   as B64
import qualified Data.Default.Class       as Def
import qualified Data.Either as Ei
import qualified Data.Text    as Tx
import qualified Data.Text.Encoding       as Tx.E
import qualified Data.X509                as X509             -- from 'x509'
import qualified Data.X509.CertificateStore     as X509.CS           -- from 'x509-store'
import qualified Network.Connection       as Net.Conn         -- from 'connection'
import qualified Network.HTTP.Client.TLS  as HTTP.Client.TLS  -- from 'http-client-tls'
import qualified Network.TLS              as Net.TLS          -- from 'tls'
import qualified Network.TLS.Extra        as Net.TLS.Ex       -- from 'tls'
import qualified System.X509              as X509             -- from 'x509-system'

import qualified Network.Wreq as W

unsafeCertificateValidation :: IO.MonadIO m => [X509.SignedExact X509.Certificate] -> m W.Options
unsafeCertificateValidation extraCerts
  = do
      systemCertStore <- IO.liftIO X509.getSystemCertificateStore

      let pinnedStore = X509.CS.makeCertificateStore extraCerts

          stores = systemCertStore <> pinnedStore

      let tlsClientParams
            = (Net.TLS.defaultParamsClient "" BS.empty)
                { Net.TLS.clientShared
                    = Def.def { Net.TLS.sharedCAStore = stores
                              }
                }

          tlsSettings = Net.Conn.TLSSettings tlsClientParams

          managerSettings  = HTTP.Client.TLS.mkManagerSettings tlsSettings Nothing

          options = W.defaults & W.manager .~ Left managerSettings
                               & W.checkStatus .~ Just (\_ _ _ -> Nothing)


      pure options

missingCertificateBase64 :: Tx.Text
missingCertificateBase64

missingCertificateBinary :: BS.ByteString
missingCertificateBinary
  = B64.decodeLenient (Tx.E.encodeUtf8 symmantecCertBase64)

main :: IO ()
main
  = do
      let eitherCert :: Either String (X509.SignedExact X509.Certificate) = X509.decodeSignedObject symmantecCertBinary
          extraCerts = Ei.either (const []) pure eitherCert

      putStrLn "Getting..."
      opts <- unsafeCertificateValidation extraCerts
      r <- W.getWith opts "vendor api url here"
      putStrLn "Got..."
      p

[vdukhovni]

TLS servers are required to return a complete chain up to, but not always including the trusted root CA.
While some servers omit required intermediate certificates, and some clients compensate for this in various ways, the servers are misconfigured.

https://tools.ietf.org/html/rfc5246#section-7.4.2

   certificate_list
      This is a sequence (chain) of certificates.  The sender's
      certificate MUST come first in the list.  Each following
      certificate MUST directly certify the one preceding it.  Because
      certificate validation requires that root keys be distributed
      independently, the self-signed certificate that specifies the root
      certificate authority MAY be omitted from the chain, under the
      assumption that the remote end must already possess it in order to
      validate it in any case.

https://tools.ietf.org/html/draft-ietf-tls-tls13-18#section-4.4.1

   certificate_list  This is a sequence (chain) of CertificateEntry
      structures, each containing a single certificate and set of
      extensions.  The sender's certificate MUST come in the first
      CertificateEntry in the list.  Each following certificate SHOULD
      directly certify one preceding it.  Because certificate validation
      requires that trust anchors be distributed independently, a
      certificate that specifies a trust anchor MAY be omitted from the
      chain, provided that supported peers are known to possess any
      omitted certificates.

[crossleydominic]

Excellent, thanks for the clarity on the specific parts of the TLS specification. I was hoping that would be the case so that we can point out that the vendors are misconfigured. I added the code example workaround in my original message just in case anybody finds themselves working with a truculent vendor and need an alternative solution. Thanks.