Custom domains do not work with pure websockets

[bensie]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
The current implementation requires that you specify a single AWS-supplied GraphQL URL in the config and then the client discovers the wss:// URL by replacing the AWS-specific "appsync-api" string with "appsync-realtime-api". These differing hostnames clearly matter as they resolve to different addresses, but since a real-time URL cannot currently be specified in the configuration, it will attempt this same "discovery" on a custom URL.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.
Create a CloudFront distribution that points to your AppSync API hostname and attempt to use subscriptions. They cannot connect because the pure websockets implementation requires connecting to the appsync-realtime-api endpoint.

What is the expected behavior?
Allow specifying a separate real-time URL in the config so custom domain subscription (websocket) connections can be made.

Which versions and which environment (browser, react-native, nodejs) / OS are affected by this issue? Did this work in previous versions?
v3.0.2

[zachboyd]

@bensie Over the past few days I worked with the new realtime endpoint (pure websocket) behind a custom domain. There were a few nuances to be aware of that would require changes to this library. In our case we use JWT token for authentication and a host value that is included in a header query string parameter needs to be the default graphql endpoint (not the realtime) or you cannot successfully authenticate. This is why you have to pass in the normal endpoint and the library then translates the actual websocket url into the realtime url. In the end, we scrapped this lib due to crashes and moved to using https://github.com/apollographql/subscriptions-transport-ws which has proven to be more reliable. This took modifications since we are now responsible for generating the header and payload query parameters that are appended to the realtime endpoint url for the connection as well as extending the lib to support a custom start_ack message that the appsync gql server sends to the client. Not sure if it will be best in the long run but there does seem to be a larger community behind it.

[bpceee]

@zachboyd That sounds awesome.
So where do you make modifications of the payload? Inside the subscriptions-transport-ws library?
Is there any chance to opensource your solution? Thanks!

[bpceee]

Hi @bensie , I guess you can try to utilize two domains. One as appsync-api.yourdomain.com another as appsync-realtime-api.yourdomain.com.

[wolfeidau]

I would love to know how you got this to work as well @zachboyd any tips would be appreciated.

[scanning]

I've had a similar experience and can confirm what @zachboyd said in regards to the following:

a host value that is included in a header query string parameter needs to be the default graphql endpoint (not the realtime) or you cannot successfully authenticate

I set up two CloudFronts with custom DNS domains, one with appsync-api in it and the other with appsync-realtime-api in it, configuring both to point at the appropriate endpoints. In the GraphQL request I noticed there is an authorisation extension that contains a ‘host’ field that appears to have to match the appsync-api url (in my case .appsync-api.us-east-1.amazonaws.com). Here is an example snippet from the subscribe payload:

{"authorization":{"Authorization":"ey...","host":".appsync-api.us-east-1.amazonaws.com"}}}

[zachboyd]

@bpceee All modifications were made outside of the library through a combination of callbacks, middleware, and extending the SubscriptionClient from subscriptions-transport-ws. We are still working on a few things and then would be happy to open-source the solution for those interested.

@scanning That is the same approach we took in regards to cloudfront.

[bpceee]

@zachboyd That's so great. Thanks!

[0xdevalias]

@zachboyd Curious if you ever managed to polish up your solution and/or release it open source? And if not, if you have a working prototype, if you're able to share the method/required changes here at all?

[zachboyd]

@0xdevalias Thanks for the reminder. It is not polished, but I went ahead and threw the relevant portions for creating the WebSocketLink into a gist for those that are interested. The goal was to not fork the existings libs so as mentioned before all modifications were made outside of the library through a combination of callbacks, middleware, and extending the SubscriptionClient from subscriptions-transport-ws. It might need tweaking based on your needs but hopefully it can get you pointed in the right direction.

https://gist.github.com/zachboyd/f5630736b0a5a9b627d61bfd25299c90

[maartenvanderhoef]

Question to everyone here, could this work as origin-request edge lambda ? I tried for a bit was hitting errors, I don't know much about WS in general.

exports.handler = (event, context, callback) => {
    const request = event.Records[0].cf.request;

    const apiDomain = 'xx-api.eu-central-1.amazonaws.com';
    const realTimeApiDomain = 'yy-realtime-api.eu-central-1.amazonaws.com';

    let destDomain = apiDomain;
    if ('sec-websocket-protocol' in request.headers) {
        destDomain = realTimeApiDomain;
    }
    console.log(request);
    request.origin = {
        custom: {
            domainName: destDomain,
            port: 443,
            protocol: 'https',
            path: '',
            sslProtocols: ['TLSv1', 'TLSv1.1', 'TLSv1.2'],
            readTimeout: 5,
            keepaliveTimeout: 5,
            customHeaders: {}
        }
    };

    request.headers['host'] = [{
        key: 'host',
        value: destDomain
    }];

    callback(null, request);
};

[razor-x]

I'm happy to share the origin-req lambda I cooked up. If the client adds the ws=true and authorization=<token> to the query string then this will change the custom origin domain to use the realtime one and add the required headers, etc.

'use strict'

process.env.NODE_ENV = 'production'

const realtimeSearchParam = 'ws'
const authorizationSearchParam = 'authorization'

const handler = (event, context, callback) => {
  const request = event.Records[0].cf.request
  const searchParams = new URLSearchParams(request.querystring)

  if (!isRealtimeReq(searchParams)) return callback(null, request)

  const { domainName } = request.origin.custom
  request.origin.custom.domainName = toRealtimeDomain(domainName)
  request.headers.host = [{ key: 'Host', value: domainName }]
  request.querystring = getQuerystring(request, searchParams)

  return callback(null, request)
}

const isRealtimeReq = (searchParams) => {
  const param = searchParams.get(realtimeSearchParam)
  return param && param.toString() === 'true'
}

const toRealtimeDomain = (domainName) =>
  domainName.replace('appsync-api', 'appsync-realtime-api')

const getQuerystring = (request, searchParams) => {
  const hostHeader = request.headers.host
  const authorization = searchParams.get(authorizationSearchParam)

  const headerObj = {}
  if (hostHeader && hostHeader[0]) headerObj.host = hostHeader[0].value
  if (authorization) headerObj.authorization = authorization
  const headerJson = JSON.stringify(headerObj)
  const headerBase64 = Buffer.from(headerJson).toString('base64')

  const payloadBase64 = Buffer.from('{}').toString('base64')

  searchParams.set('header', headerBase64)
  searchParams.set('payload', payloadBase64)
  searchParams.delete(realtimeSearchParam)
  searchParams.delete(authorizationSearchParam)

  return searchParams.toString()
}

exports.handler = handler

[maartenvanderhoef]

@razor-x Could you help me understand the following:
With using the amplify library, the client sends the header-query param with WS request already, albeit with a 'wrong' endpoint. Would it be possible to deconstruct the header, fix the host in it, and pass the newly created header on, or will subsequent requests without the header fail ?

[arcanereinz]

Unfortunately an origin-request lambda@edge won't work with cloudfront since cloudfront does not support the Connection or Upgrade headers needed to convert a https connection to a wss (websocket) connection. These headers are explicitly blacklisted and cannot be sent by neither cloudfront nor lambdas (https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-requirements-limits.html#lambda-blacklisted-headers).

I believe one way around this is to use another another library as mentioned above. However I'd like to see if there's a way to put a load balancer in front of AppSync that can redirect base on query and supports websock upgraded connections; and also hard-code the x-api-key so you can do blue/green deployments without changing the client.

[danielblignaut]

Hey,

@arcanereinz your comment is misleading. Whilst those headers are not supported in Lambda@edge functions or in your cache key but they are still successfully passed through cloudfront to appsync transiently. I can confirm that you CAN successfully foward on the connection handshake request using cloudfront, and an origin request lambda@edge similar to @razor-x . The real problem though is that once the handshake is approved and your connection is upgraded to operate over websockets, there is no way to intercept the websocket messages using lambda@edge in cloudfront which means messages sent for "subscription registration" (see https://docs.aws.amazon.com/appsync/latest/devguide/real-time-websocket-client.html#subscription-registration-message ) contain the incorrect "host" leading to an error.

If anyone is aware of how to use lambda@edge or cloudfront functions to intercept websocket messages and adjust the payload body, please let me know. That would be the final step needed to support Appsync subscriptions without having to worry about hardoding / adding an env variable onto your front end client which is really a problem for us.

[ankon]

A +1/data point: We were able to use @zachboyd 's gist (https://gist.github.com/zachboyd/f5630736b0a5a9b627d61bfd25299c90, thank you so much!) to replace aws-appsync-subscription-link in our Apollo v3 app.

What we did:

1. Create a CloudFront distribution with /graphql pointing to the regular appsync-api endpoint, and /graphql-ws pointing to the appsync-realtime-api endpoint¹
2. Expose the CloudFront distribution under our own domain
3. Provide configuration for the application: the regular and realtime endpoints (custom domains) and the host for the regular API
4. Configure the Apollo client following https://www.apollographql.com/docs/react/data/subscriptions/ using a "split" link for HTTP/WS

---

¹: If you need to do this in CloudFormation, enjoy: !Join [ '.', [ !Select [ 0, !Split [ '.', !Select [ 2, !Split ['/', !Sub '${GraphQlApi.GraphQLUrl}' ] ] ] ], !Sub 'appsync-realtime-api.${AWS::Region}.amazonaws.com' ] ]

[onlybakam]

AppSync now supports custom domain names. See details in this blog post and visit the docs