[error] Unable to fetch events from AWS. Reason: {:option, :server_only, :honor_cipher_order}

[noozo]

Hi all, getting this error when running a broadway app inside a docker container on AWS fargate. All the AWS credentials are being correctly set and passed to the producer:

Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producers: [
        default: [
          module: {
            BroadwaySQS.Producer,
            attribute_names: :all,
            message_attribute_names: :all,
            queue_name: System.get_env("QUEUE_NAME"),
            config: get_aws_credentials(),
            receive_interval: 1000
          }
        ]
      ],
      processors: [
        default: []
      ],
      batchers: [
        default: [
          batch_size: 10,
          batch_timeout: 2000
        ]
      ]
    )

Where get_aws_credentials() is:

def get_aws_credentials do
    region = System.get_env("AWS_REGION")

    [
      secret_access_key: System.get_env("AWS_SECRET_ACCESS_KEY"),
      access_key_id: System.get_env("AWS_ACCESS_KEY_ID"),
      region: region,
      host: "sqs.#{region}.amazonaws.com",
      scheme: "https"
    ]
  end

All those env vars are present and correct inside the container.

I do notice that the queue url appears to not be constructed correctly (i think it's missing the AWS account id):

It's https://sqs.eu-west-1.amazonaws.com/my_queue, when it should be https://sqs.eu-west-1.amazonaws.com/12345678/my_queue (where 12345678 is the account id).

For completion this account has roles to be able to access test and production environments, so that could be partially the cause. Is there a setting to explicitly pass the account id to the producer?

Cheers!

[msaraiva]

Hi @nocivus!

Does it work if you pass the full queue URL instead of the name? If so, this is a known issue regarding a recent change in ex_aws_sqs. You can see more details here #30. Please let me know in case it's not related at all.

[noozo]

Hi @msaraiva, still happens if i set the host with the account id. The message then becomes:
[warn] ExAws: HTTP ERROR: {:option, :server_only, :honor_cipher_order} for URL: "https://sqs.eu-west-1.amazonaws.com/12345678/my_queue" ATTEMPT: 10 :(

EDIT: Did not attempt to put the full url in the queue_name, will try that.

[noozo]

Seems to be unrelated, tbh.

In local development it works properly, only fails when running inside an alpine container on AWS.

If i put the full url in the queue name it fails locally (as it should, since it's just appending that name to the host).

[chris-brace]

This is currently an issue with hackney. It could have to do with lack of verison pinning. OTP updated to 22.1 and changed a return type. For now replace your hackney dep with {:hackney, github: "benoitc/hackney", override: true}

benoitc/hackney#591

[noozo]

thanks @chris-brace, will give that a shot...

[chris-brace]

No problem. However, if you also upgrade from broadway_sqs 0.2.0 to 0.3.0 you'll need to set the ENTIRE queue url as the queue_name param. That's discussed in my issue @msaraiva linked.

[noozo]

Ah, then i'll probably upgrade that. Btw, with 0.2.0 i still get the same error even after updating hackney to master. Do you think it could have anything to do with the fact that i use this docker image to build:

# ---- Build Stage ----
FROM elixir:alpine AS app_builder

and this one to actually run the released version?

FROM alpine:3.9 AS app

Btw, reason i use 3.9 is because the app does not even start (seg fault) with alpine latest.

[chris-brace]

This is 100% what caused it. c.f.: c0b/docker-elixir#122 (comment)

I fixed this by changing the hackney dep to point at master (really annoying, might consider switching to mojito). Just don't forget to add

RUN apk add --no-cache git

to your build stage because mix requires git to pull from the repo.

[noozo]

Doesn't seem to work, maybe i need to wait for the new hackney version:

[chris-brace]

Oh boy.. yeah so i would recommend resolving the dep overrides and stuff locally. Are you using an umbrella project?

edit: Btw they've created a serious mess here, it's not you :) Just remember this next time people start talking about how x package management (npm, etc) system is a disaster. They're all like this.

[noozo]

Yeah, trying to sort it out, let's see how it goes. I might end up waiting for hackney to have a version out. Thanks for the help!

[ixtli]

They need to make a point version. Dunno how long that's gonna take. If you're using an umbrella project you can just put the dependency with override: true in the umbrella mix.exs and that SHOULD take precedence over everything. Also you may be able to just swap out hackney and replace it with mojito though i haven't tried this.

[noozo]

Ended up solving it by explicitly setting the cowboy version i needed in my mix file:
{:cowboy, "~> 2.5", override: true},. Also bumped broadway_sqs to 0.3.0 and set hackney to github master.

Thanks for the help, guys!

[noozo]

Work around works :)