github actions SSL error

[michelson]

Hello there, I'm using a cldr library (ex_cldr_dates_times) which downloads some data from raw.githubusercontent, this works on development but when I run this on Github actions I get an SSL error:

Error: 05:33:08.872 [error] Failed to connect to 'raw.githubusercontent.com' to download locale :pt. Reason: {:tls_alert, {:unexpected_message, 'TLS client: In state hello_middlebox_assert at ssl_gen_statem.erl:736 generated CLIENT ALERT: Fatal - Unexpected Message\n {unexpected_msg,{internal,{encrypted_extensions,\#{sni => {sni,[]}}}}}'}}

here is the build with the error: https://github.com/rauversion/rauversion-phx/runs/7912634783?check_suite_focus=true

is there a way to bypass this installation?

[kipcole9]

@michelson Thanks for the report and sorry for the inconvenience. Having the locale data available is critical and unavoidable - its the source of data that drives the whole ex_cldr ecosystem.

One way to possibly avoid this is if you check the locales into your repo (the default directory where locales will be located is :code.priv_dir(otp_app) or :code.priv_dir(:ex_cldr).

It looks like this is an issue on Github's side and something I can't control. On the other hand I've never see this issue reported either so I'm unlikely to be able to resolve it on my side (I will however do some googling).

[kipcole9]

One other thought, since its reporting an SSL handshake error (and I really have no domain expertise in this area), is to include either {:certifi, "~> 2.9"} or {:castore, "~> 0.1.18"} as dependencies. The locale installer will use the certificates from these in preference to the built-in certificates on your host machine.

[LostKobrakai]

Alternatively you could try explicitly updating the OS level certificates.

[michelson]

Hi! thanks for the quick reply, I've added the {:certifi, "> 2.9"} or {:castore, "> 0.1.18"} dependences but I get the same error, also I've updated the system certs in ubuntu with sudo apt-get install ca-certificates -y but didn't solve the issue.

One way to possibly avoid this is if you check the locales into your repo (the default directory where locales will be located is :code.priv_dir(otp_app) or :code.priv_dir(:ex_cldr).

How can I check the locales in the repo? I've done the following but the locales are not checked out in the repo:

config :ex_cldr,
  default_locale: "en",
  default_backend: Rauversion.Cldr,
  json_library: Jason,
  data_dir: "./priv/cldr",
  force_locale_download: false

[tdtadeu]

We ran into the same issue and apparently updating the project to OTP 25 solved it and unblocked our CI pipeline.

Is this something you can consider doing?

[LostKobrakai]

https://elixirforum.com/t/github-ci-down-is-it-working-for-you/49661 maybe there's some issue/changes on github's side?

[michelson]

Thank you all. Upgrade to opt 25 did solve the issue. Super weird, though.

[kipcole9]

Thanks for the update @michelson; I'll add some notes to the readme in case other people have this issue.

[sergiotapia]

We had to make this change to unclog the CI on Github. Hope this helps people.

--- otp: ['24.3.4']
+++ otp: ['25.0.4']

[WillRaben]

I'm experiencing this exact error when building using the official debian bullseye image for elrang 24.3.4; I'm looking for a workaround to avoid upgrading to OTP 25.
I've manually copied the locales from a successful compile to the container's deps/ex_cldr-2.23.0/priv/
It does seem to compile by account of not trying to hit github through httpc, but then another app that has ex_cldr in its dependencies, fails to compile.

From what I could gather the underlying issue could be something that the host OS is missing, some configuration that makes TLS fail. I tried from an ubuntu 20.04.3 box and had no issues.

[kipcole9]

@WillRaben, I think if you add the :data_dir option to the global config in config.exs then all the libraries will share that configuration (although I will say that not 100% certain since this is hard to test properly).

I am working on an env var to disable TLS checking as an option and should have that release out in the next couple of hours. It seems a reasonable thing to help this situation in CI.

[kipcole9]

I have published ex_cldr version 2.33.1-rc.0 that changes HTTPS request handling slightly. In particular it applies the ERLF guidelines around cypher configuration. Based on the error message that may be the issue so any feedback would be welcome.

Next step, if required, is to add an CLDR_UNSAFE_HTTPS environment variable to dictate whether to do hostname checking (given the the error was related to sni verification).

[WillRaben]

@kipcole9
The release candidate, that changes the httpc request handling, fixes the compile issue without needing to upgrade to OTP 25

By replacing
{:ex_cldr, "~> 2.0"}

With:
{:ex_cldr, git: "https://github.com/elixir-cldr/cldr.git", tag: "v.2.33.1-rc.0", override: true},

override: true was needed because of a dependency in the package ex_cldr_territories that has {:ex_cldr, "~> 2.0"} in its mix.exs.

Only outliers in the compilation (most likely because of overriding the child dependency) are two instances of warnings due to deprecation:

warning: Cldr.Config.known_locale_names/1 is deprecated. Use Cldr.Locale.Loader.known_locale_names/1
warning: Cldr.Config.get_locale/2 is deprecated. Use Cldr.Locale.Loader.get_locale/2

Upon some more testing, seems like the issue might be related to fly.io and similar providers, would be nice to know the context of @michelson 's build. I replicated a docker build locally and had no issue with httpc hitting github from a container using the official erlang 24.3.4 docker image running on ubuntu 20.04.3.

Haven't found any other side effects. Will definitely let you know if something else comes up.

Massive thanks for the swift reply.

[michelson]

Hi @WillRaben, the context of the build: OTP 24 and Elixir 13
This was the failing build: https://github.com/rauversion/rauversion-phx/runs/7912634783?check_suite_focus=true
Let me know if you need to know any additional information.

[kipcole9]

@WillRaben, using a GitHub branch as a dependencies won't actually download anything since the locales are all in the repo. To test whether this is a solution your deps should look like:

def deps do
  [
    ex_cldr: "~> 2.33.1-rc", override: true,
    ...
  ]
end

That will then certainly show if downloading is now working on the GitHub CI platform.

[kipcole9]

The deprecations most like coming from ex_cldr_territories if it hasn't been updated to change the calls that raise the warnings. If that's the case I'll work with @Schultzer to update ex_cldr_territories to remove the deprecations. The deprecations are "harmless" for the purposes of testing the locale installation issue.

You could try: mix deps.update ex_cldr_territories to see if there is a later version, I'm not sure what version you are currently using.

[WillRaben]

@WillRaben, using a GitHub branch as a dependencies won't actually download anything since the locales are all in the repo. To test whether this is a solution your deps should look like:

def deps do
  [
    ex_cldr: "~> 2.33.1-rc", override: true,
    ...
  ]
end

That will then certainly show if downloading is now working on the GitHub CI platform.

Ah, I see, TIL. No wonder the build got noticeably bigger.

I just ran the build without the github branch and unfortunately, the error returned and it failed to compile.

#46 167.9 22:27:10.879 [notice] TLS :client: In state :hello_middlebox_assert at ssl_gen_statem.erl:736 generated CLIENT ALERT: Fatal - Unexpected Message
#46 167.9 - {:unexpected_msg, {:internal, {:encrypted_extensions, %{sni: {:sni, []}}}}}
#46 168.0 22:27:10.971 [error] Failed to connect to 'raw.githubusercontent.com' to download 'https://raw.githubusercontent.com/elixir-cldr/cldr/v2.33.1-rc.0/priv/cldr/locales/fr.json'

Regarding ex_cldr_territories, ours is: {:ex_cldr_territories, "~> 2.2"} so that's probably why it's still using deprecated calls, current version is 2.4. I will update and keep an eye for any warnings.

[kipcole9]

Thanks for the update. I will implement the environment variable later today my time as an escape hatch (may take up to 18 hours for me to get to this and publish).

…

Sent from my iPhone

On 21 Aug 2022, at 08:38, Will Raben ***@***.***> wrote:  @WillRaben, using a GitHub branch as a dependencies won't actually download anything since the locales are all in the repo. To test whether this is a solution your deps should look like: def deps do [ ex_cldr: "~> 2.33.1-rc", override: true, ... ] end That will then certainly show if downloading is now working on the GitHub CI platform. Ah, I see, TIL. No wonder the build got noticeably bigger. I just ran the build without the github branch and unfortunately, the error returned and it failed to compile. #46 167.9 22:27:10.879 [notice] TLS :client: In state :hello_middlebox_assert at ssl_gen_statem.erl:736 generated CLIENT ALERT: Fatal - Unexpected Message #46 167.9 - {:unexpected_msg, {:internal, {:encrypted_extensions, %{sni: {:sni, []}}}}} #46 168.0 22:27:10.971 [error] Failed to connect to 'raw.githubusercontent.com' to download 'https://raw.githubusercontent.com/elixir-cldr/cldr/v2.33.1-rc.0/priv/cldr/locales/fr.json' Regarding ex_cldr_territories, ours is: {:ex_cldr_territories, "~> 2.2"} so that's probably why it's still using deprecated calls, current version is 2.4. I will update and keep an eye for any warnings. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[kipcole9]

cldr_utils version 2.19.0 now supports the environment variable CLDR_UNSAFE_HTTPS. The changelog entry reads:

Enhancements

• Sets SNI option for SSL connections

• Supports CLDR_UNSAFE_HTTPS environment variable option which, if set to anything other than FALSE, false, nil or NIL will not perform peer verification for HTTPS requests. This may be used in circumstances where peer verification is failing but if generally not recommended.

Configuration

To use this option, the following configuration should be used:

def deps do
  [
    # Make sure that 2.33.1 is resolved
    # `mix deps.update ex_cldr cldr_utils` will also work
    {:ex_cldr, "~> 2.33"},
    {:cldr_utils, "~> 2.19"},
    ...
  ]
end

Hopefully this will let you get past whatever the issue is in GitHub CI until we can get to the bottom of it.

[WillRaben]

@kipcole9 Thanks for staying on top this.

Ok, so I've tested the CLDR_UNSAFE_HTTP approach with our umbrella app, and I am very confused by the results:

In the builder(Fly.io), I am getting this logged out from mix deps.compile:

...
#47 83.92 Generated timex app
#47 84.02 ==> cldr_utils
#47 84.02 Compiling 11 files (.ex)
#47 84.90 Generated cldr_utils app
#47 85.01 ==> ex_cldr
#47 85.01 Compiling 1 file (.yrl)
#47 85.05 Compiling 1 file (.xrl)
#47 85.11 Compiling 2 files (.erl)
#47 85.48 Compiling 41 files (.ex)
#47 105.5 Compiling lib/cldr/validity/subdivision.ex (it's taking more than 10s)
#47 117.8 Generated ex_cldr app
#47 117.8 ==> ex_cldr_territories
#47 117.8 Compiling 3 files (.ex)
#47 118.5 Generated ex_cldr_territories app
#47 118.6 ==> mogri ...

But then once one of the apps that requires ex_cldr starts compiling, it looks like it tries to hit github again without going through cldr_utils.

...
#47 173.0 ===> Compiling cowboy_telemetry
#47 173.8 ==> smarty_streets
#47 173.8 Compiling 10 files (.ex)
#47 174.7
#47 174.7 16:41:19.711 [notice] TLS :client: In state :hello_middlebox_assert at ssl_gen_statem.erl:736 generated CLIENT ALERT: Fatal - Unexpected Message
#47 174.7  - {:unexpected_msg, {:internal, {:encrypted_extensions, %{sni: {:sni, []}}}}}
#47 174.7
#47 174.7 16:41:19.763 [error] Failed to connect to 'raw.githubusercontent.com' to download 'https://raw.githubusercontent.com/elixir-cldr/cldr/v2.33.1/priv/cldr/locales/es.json'
#47 174.7
#47 174.7 16:41:19.774 [notice] TLS :client: In state :hello_middlebox_assert at ssl_gen_statem.erl:736 generated CLIENT ALERT: Fatal - Unexpected Message
#47 174.7  - {:unexpected_msg, {:internal, {:encrypted_extensions, %{sni: {:sni, []}}}}}
#47 174.7
#47 174.7 16:41:19.775 [error] Failed to connect to 'raw.githubusercontent.com' to download 'https://raw.githubusercontent.com/elixir-cldr/cldr/v2.33.1/priv/cldr/locales/fr.json'
#47 174.7 Generating SmartyStreets.Core.CldrHelper for 4 locales named [:en, :es, :fr, :und] with a default locale named :en
#47 174.8
#47 174.8 == Compilation error in file lib/core/Cldr_helper.ex ==
#47 174.8 ** (RuntimeError) The locale file for :es was not found.
#47 174.8     (ex_cldr 2.33.1) lib/cldr/config/config.ex:1582: Cldr.Config.locale_path!/2
#47 174.8     (stdlib 3.17.2) erl_eval.erl:685: :erl_eval.do_apply/6
#47 174.8     (stdlib 3.17.2) erl_eval.erl:893: :erl_eval.expr_list/6
#47 174.8     (stdlib 3.17.2) erl_eval.erl:408: :erl_eval.expr/5
#47 174.8     (stdlib 3.17.2) erl_eval.erl:123: :erl_eval.exprs/5
#47 174.8     (elixir 1.13.4) lib/enum.ex:2396: Enum."-reduce/3-lists^foldl/2-0-"/3
#47 174.8     (stdlib 3.17.2) erl_eval.erl:685: :erl_eval.do_apply/6
#47 174.8 could not compile dependency :smarty_streets, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile smarty_streets", update it with "mix deps.update smarty_streets" or clean it with "mix deps.clean smarty_streets"
------
Error failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c mix deps.compile]: exit code: 1
...

The contents of lib/core/Cldr_helper.ex

defmodule SmartyStreets.Core.CldrHelper do
  use Cldr,
      locales: ["en", "es", "fr"],
      default_locale: "en",
      supress_warnings: true
end

dependencies listed on the app that fails to compile:

defp deps do
    [
      {:httpoison, "~> 1.6"},
      {:poison, "~> 3.0"},
      {:cldr_utils, "~> 2.19"},
      {:ex_cldr, "~> 2.33"},
      {:ex_cldr_territories, "~> 2.4"}
    ]
  end

Builder has: ENV CLDR_UNSAFE_HTTPS="true" set and verified working by adding an RUN echo $CLDR_UNSAFE_HTTPS just before RUN mix deps.compile

For the moment, we just got the whole repo so the build on fly.io passed, but this created a problem with gigalixir where it fails with: Slug size (680MB) can not exceed 512MB when we tried it. Although gigalixir doesn't error when trying to download from github, we did it just as a test. No need to get the whole repo there.

Could this specific failure( not using cldr_utils) be due to additional configuration needed for umbrella applications?

I wonder what, if anything, is missing from that debian bullseye buildpack that the official Erlang 24 image uses, that's making peer verification fail.

[kipcole9]

Would you mind running mix hex.outdated and paste the results? It sounds like ex_cldr is still at 2.33 not 2.33.1 but this is the quickest way to find out.

I do want to find the root cause, but first priority is to get you back running smoothly.

[WillRaben]

I've temporarily added mix hex.outdated --all to the build steps right after deps.get, like so:

COPY . /app/
RUN mix deps clean --all
RUN mix deps.get  --only prod
RUN mix hex.outdated --all
RUN mix deps.compile

Here's the output from hex.outdated --all:

> [build 33/41] RUN mix hex.outdated --all:
#46 2.622 Dependency              Current  Latest   Status
#46 2.622 bamboo                  1.6.0    2.2.0    Update not possible
#46 2.622 bamboo_gmail            0.2.0    0.2.0    Up-to-date
#46 2.622 barlix                  0.6.1    0.6.1    Up-to-date
#46 2.622 bcrypt_elixir           2.3.0    3.0.1    Update not possible
#46 2.622 broadway                1.0.1    1.0.3    Update possible
#46 2.622 bunt                    0.2.0    0.2.0    Up-to-date
#46 2.622 certifi                 2.9.0    2.9.0    Up-to-date
#46 2.622 cldr_utils              2.19.0   2.19.0   Up-to-date
#46 2.622 combine                 0.10.0   0.10.0   Up-to-date
#46 2.622 comeonin                5.3.2    5.3.3    Update possible
#46 2.622 connection              1.1.0    1.1.0    Up-to-date
#46 2.622 cowboy                  2.9.0    2.9.0    Up-to-date
#46 2.623 cowboy_telemetry        0.3.1    0.4.0    Update possible
#46 2.623 cowlib                  2.11.0   2.11.0   Up-to-date
#46 2.623 credo                   1.6.4    1.6.6    Update possible
#46 2.623 db_connection           2.4.0    2.4.2    Update possible
#46 2.623 decimal                 2.0.0    2.0.0    Up-to-date
#46 2.623 dialyxir                1.0.0    1.2.0    Update possible
#46 2.623 earmark_parser          1.4.13   1.4.26   Update possible
#46 2.623 ecto                    3.6.2    3.8.4    Update not possible
#46 2.623 ecto_enum               1.4.0    1.4.0    Up-to-date
#46 2.623 ecto_sql                3.6.2    3.8.3    Update possible
#46 2.623 elixir_make             0.6.2    0.6.3    Update possible
#46 2.623 elixir_uuid             1.2.1    1.2.1    Up-to-date
#46 2.623 eqrcode                 0.1.10   0.1.10   Up-to-date
#46 2.623 erlex                   0.2.6    0.2.6    Up-to-date
#46 2.623 ex_aws                  2.2.3    2.3.4    Update possible
#46 2.623 ex_aws_kms              2.2.0    2.2.0    Up-to-date
#46 2.623 ex_aws_s3               2.2.0    2.3.3    Update possible
#46 2.623 ex_aws_s3_crypto        2.0.1    3.0.1    Update possible
#46 2.623 ex_cldr                 2.33.1   2.33.1   Up-to-date
#46 2.623 ex_cldr_currencies      2.11.0   2.14.1   Update possible
#46 2.623 ex_cldr_numbers         2.19.0   2.27.2   Update possible
#46 2.623 ex_cldr_territories     2.4.0    2.4.0    Up-to-date
#46 2.623 ex_doc                  0.24.2   0.28.5   Update possible
#46 2.623 excoveralls             0.14.0   0.14.6   Update possible
#46 2.623 file_size               3.0.1    3.0.1    Up-to-date
#46 2.623 file_system             0.2.10   0.2.10   Up-to-date
#46 2.623 floki                   0.31.0   0.33.1   Update possible
#46 2.623 flow                    1.1.0    1.2.0    Update possible
#46 2.623 gen_stage               1.1.2    1.1.2    Up-to-date
#46 2.623 gettext                 0.20.0   0.20.0   Up-to-date
#46 2.623 goth                    1.1.0    1.3.1    Update not possible
#46 2.623 hackney                 1.18.1   1.18.1   Up-to-date
#46 2.623 hash_color_avatar       0.1.0    0.1.0    Up-to-date
#46 2.623 heroicons               0.2.1    0.3.2    Update possible
#46 2.623 html_entities           0.5.2    0.5.2    Up-to-date
#46 2.623 httpoison               1.8.0    1.8.2    Update possible
#46 2.623 idna                    6.1.1    6.1.1    Up-to-date
#46 2.623 inflex                  2.1.0    2.1.0    Up-to-date
#46 2.623 jason                   1.3.0    1.3.0    Up-to-date
#46 2.623 joken                   2.3.0    2.5.0    Update possible
#46 2.623 jose                    1.11.1   1.11.2   Update possible
#46 2.623 machinery               1.0.0    1.0.0    Up-to-date
#46 2.623 mail                    0.2.2    0.2.3    Update possible
#46 2.623 makeup                  1.0.5    1.1.0    Update possible
#46 2.623 makeup_elixir           0.15.1   0.16.0   Update possible
#46 2.623 makeup_erlang           0.1.1    0.1.1    Up-to-date
#46 2.624 meck                    0.9.2    0.9.2    Up-to-date
#46 2.624 metrics                 1.0.1    2.5.0    Update not possible
#46 2.624 mime                    1.6.0    2.0.3    Update not possible
#46 2.624 mimerl                  1.2.0    1.2.0    Up-to-date
#46 2.624 mix_test_watch          1.0.3    1.1.0    Update possible
#46 2.624 mock                    0.3.7    0.3.7    Up-to-date
#46 2.624 mogrify                 0.9.1    0.9.2    Update possible
#46 2.624 neuron                  5.0.0    5.0.0    Up-to-date
#46 2.624 nimble_csv              1.2.0    1.2.0    Up-to-date
#46 2.624 nimble_options          0.3.7    0.4.0    Update not possible
#46 2.624 nimble_parsec           1.1.0    1.2.3    Update possible
#46 2.624 number                  1.0.3    1.0.3    Up-to-date
#46 2.624 oauther                 1.3.0    1.3.0    Up-to-date
#46 2.624 parse_trans             3.3.1    3.4.1    Update not possible
#46 2.624 pdf_generator           0.6.2    0.6.2    Up-to-date
#46 2.624 phoenix                 1.5.9    1.6.11   Update not possible
#46 2.624 phoenix_active_link     0.2.1    0.3.2    Update possible
#46 2.624 phoenix_ecto            4.3.0    4.4.0    Update possible
#46 2.624 phoenix_html            2.14.3   3.2.0    Update not possible
#46 2.624 phoenix_live_dashboard  0.4.0    0.6.5    Update possible
#46 2.624 phoenix_live_reload     1.3.2    1.3.3    Update possible
#46 2.624 phoenix_live_view       0.15.7   0.17.11  Update not possible
#46 2.624 phoenix_pubsub          2.0.0    2.1.1    Update possible
#46 2.624 phx_gen_auth            0.7.0    0.7.0    Up-to-date
#46 2.624 plug                    1.10.4   1.13.6   Update not possible
#46 2.624 plug_cowboy             2.5.0    2.5.2    Update possible
#46 2.624 plug_crypto             1.2.3    1.2.3    Up-to-date
#46 2.624 png                     0.2.1    0.2.1    Up-to-date
#46 2.624 poison                  3.1.0    5.0.0    Update not possible
#46 2.624 postgrex                0.15.9   0.16.4   Update not possible
#46 2.624 ranch                   1.8.0    2.1.0    Update not possible
#46 2.624 short_uuid              2.1.0    2.1.0    Up-to-date
#46 2.624 ssl_verify_fun          1.1.6    1.1.6    Up-to-date
#46 2.624 sweet_xml               0.6.6    0.7.3    Update possible
#46 2.624 telemetry               0.4.3    1.1.0    Update not possible
#46 2.624 telemetry_metrics       0.6.0    0.6.1    Update possible
#46 2.624 telemetry_poller        0.5.1    1.0.0    Update not possible
#46 2.625 timex                   3.7.5    3.7.9    Update possible
#46 2.625 tzdata                  1.1.0    1.1.1    Update possible
#46 2.625 unicode_util_compat     0.7.0    0.7.0    Up-to-date
#46 2.625 xml_builder             2.2.0    2.2.0    Up-to-date
#46 2.625 zarex                   1.0.1    1.0.3    Update possible
#46 2.833
#46 2.833 Run `mix hex.outdated APP` to see requirements for a specific dependency.
#46 2.833
#46 2.833 To view the diffs in each available update, visit:
#46 2.833 https://hex.pm/l/qXBtZ

Looks like it's picked up correctly 2.33.1

[kipcole9]

Nothing is downloaded until your SmartyStreets.Core.CldrHelper is compiled and that's where the error is occurring. And I'm out of ideas since it appears the issue isn't related to verifying the peer but something more fundamental in the networking stacks of either erlang or Ubuntu and debugging that is well beyond my ability.

I is possible to statically include the locales in your repo so you don't have to compile from GitHub, I'll update this issue with how to do that separately (after I verify the configuration, its not that common and I haven't used it in a while!)

[kipcole9]

@voltone, please accept my apologies for invoking your name but I'm struggling to even find a direction to investigate with this issue.

The issue seems to be related to Ubuntu, Github Actions and some versions (not OTP 25) of the :ssl app. The issue is resolved by upgrading to OTP 25 but not all users can do that. The issue does not appear to occur on development machines or production - only in Github Actions.

During compilation, locale data files are downloaded from GitHub itself and used to generate a lot of functions. This is the error:

Error: 05:33:08.872 [error] Failed to connect to 'raw.githubusercontent.com' to download locale :pt. Reason: {:tls_alert, {:unexpected_message, 'TLS client: In state hello_middlebox_assert at ssl_gen_statem.erl:736 generated CLIENT ALERT: Fatal - Unexpected Message\n {unexpected_msg,{internal,{encrypted_extensions,\#{sni => {sni,[]}}}}}'}}

If you have any guidance about understanding what this message is suggesting and where I can dig further I'd be hugely grateful.

• The issue happens even when there is verify: :none.

• The code that makes the request is here and follows the ERLEF recommendations which I suspect you wrote.

[WillRaben]

Nothing is downloaded until your SmartyStreets.Core.CldrHelper is compiled and that's where the error is occurring. And I'm out of ideas since it appears the issue isn't related to verifying the peer but something more fundamental in the networking stacks of either erlang or Ubuntu and debugging that is well beyond my ability.

I is possible to statically include the locales in your repo so you don't have to compile from GitHub, I'll update this issue with how to do that separately (after I verify the configuration, its not that common and I haven't used it in a while!)

Providing the actual locale files so it doesn't download seemed like the sane thing to do to me too, specially in the face of updating to OTP 25 being an easy fix (at least in our case, because we are on OTP 24.3.4, we just need to run some more tests). I just couldn't figure out what the proper folder needed to be. I tried smarty_streets/priv/locales and smarty_streets/priv, also in the ex_cldr directory too.

To expand on the context of our instance of the error:

• Umbrella application with 3 individual apps that have ex_cldr as dependency.
• Running on Fly.io using their builders to compile and build a mix release, github actions is not involved in this instance.
• Using The official Erlang 24.x debian bullseye image as base for both the build stage and runner containers.
• From what I could gather, Fly.io themselves uses nomad to manage firecracker vms which are ultimately the docker host for our debian containers. It might be a good idea for me to actually reach out to them to see if they have any insights.

Thanks for all the help everybody, much appreciated.

[kipcole9]

@WillRaben the following configuration stores the locales in a user-defined place. I would recommend you set that place to a directory that all dependent apps and all umbrella apps can point to and make sure that directory is committed as part of the repo.

defmodule DemoCldrLocalesDir do
  @locales_dir Path.expand(System.get_env("CLDR_LOCALES_DIR") || raise(ArgumentError, "No CLDR_LOCALES_DIR is set"))

  use Cldr,
    locales: ["en", "de", "fr"],
    default_locale: "en",
    data_dir:  @locales_dir |> IO.inspect(label: "Downloaded locales are stored in"),
    providers: []

end

[voltone]

Error: 05:33:08.872 [error] Failed to connect to 'raw.githubusercontent.com' to download locale :pt. Reason: {:tls_alert, {:unexpected_message, 'TLS client: In state hello_middlebox_assert at ssl_gen_statem.erl:736 generated CLIENT ALERT: Fatal - Unexpected Message\n {unexpected_msg,{internal,{encrypted_extensions,\#{sni => {sni,[]}}}}}'}}

This may be related to erlang/otp#6037, in which case it should also have been fixed in 24.3.4.1.

Alternatively, this particular scenario appears to relate to a TLS 1.3 handshake. You can limit the handshake to 1.2 by adding versions: [:"tlsv1.2"] to your ssl options, which you may want to do on OTP <25, since there were quite a few 1.3 interop issues in those older versions.

Hope that helps.

[kipcole9]

Super helpful @voltone, thank you very much indeed.

[kipcole9]

@WillRaben I have pushed this commit to restrict the TLS version to 1.2 on OTP releases < 25. Fingers crossed. Its a small library so perhaps you can try this from GitHub:

def deps do
  [
    {:cldr_utils, github: "elixir-cldr/cldr_utils", override: true},
    ...
  ]
end

Let me know if that moves us forward?

[WillRaben]

@WillRaben I have pushed this commit to restrict the TLS version to 1.2 on OTP releases < 25. Fingers crossed. Its a small library so perhaps you can try this from GitHub:

def deps do
  [
    {:cldr_utils, github: "elixir-cldr/cldr_utils", override: true},
    ...
  ]
end

Let me know if that moves us forward?

I am happy to report that this allowed the build to complete successfully. Reduced about 280mb from the resulting image. Thanks!

Would be great to hear for the github actions end of this issue, looks like it would fix things there too.

As a side note, I tried the approach you suggested with the additional configuration and having the previously downloaded locales placed in /app/locales folder and it still tried to download from github.

[kipcole9]

Great to hear @WillRaben, I've published cldr_utils version 2.19.1 with this fix so you are good to revert to using Hex. Closing now although I am confused as to why the configuration approach didn't work.

[WillRaben]

Not sure why the configuration approach failed either. It was pretty straight forward. I changed SmartyStreets.Core.CldrHelper to include the necessary code that you provided. Set the env variable to point to /app/locales folder, copied the downloaded locales there and it still tried to reach github.

It did pick the new config up when compiling though, because I saw the IO.inspect message reporting the folder.

When I was about to dig deeper into why it wasn't working, you posted the cldr_utils hotfix that limited tls to 1.2 so I dropped the config approach. It has been working flawlessly.

Thanks for all the help.

[setop]

I think I have a workaround.

I'm in elixir 1.13 (Erlang/OTP 24) and I need ex_cldr 2.25.0.

After reading this, I changed the dependency in mix.exs to point to gihhub instead of hex :

-      {:ex_cldr, "~> 2.0"},
+      {:ex_cldr, git: "https://github.com/elixir-cldr/cldr.git", tag: "v2.25.0", override: true},`

So the system - git command on alpine - contacts github and not Erlang.

It make my application compile. I haven't done a full non-regression test campaign yet.

[kipcole9]

Since installing from GitHub will download the whole repo (which is not a small one!), including all the locales, it will work because it won't ever try to download locales from GitHub. So definitely ok as a working solution as long as you recognise the payload from GitHub is substantial!

[kipcole9]

I'd also be very interested to know what requires that you use 2.25.0? If there is a regression on later releases that prevents you updating then I would definitely want to fix that.

[setop]

Since installing from GitHub will download the whole repo (which is not a small one!), including all the locales, it will work because it won't ever try to download locales from GitHub. So definitely ok as a working solution as long as you recognise the payload from GitHub is substantial!

True ! 275MB and 3 minutes to git clone.

So I made an other workaround : download the tar from github, untar and remove locales I don't need.

Then the dependency become : {:ex_cldr, path: "cldr-2.25.0", override: true}.

Saved me 2:59 minutes and 250MB :-)

[setop]

I'd also be very interested to know what requires that you use 2.25.0? If there is a regression on later releases that prevents you updating then I would definitely want to fix that.

Well, it seems that at some point (after 2.25 and before 2.33), the type of a locale has changed from string to atom. This is quite impactful on my code. I'll change that in a later version of my app so I can use up to date libraries.

[kipcole9]

Well, it seems that at some point (after 2.25 and before 2.33), the type of a locale has changed from string to atom. This is quite impactful on my code. I'll change that in a later version of my app so I can use up to date libraries.

Apologies for making extra work for you. That was the "last" step in normalising the fields of the Cldr.LanguageTag.t struct to be atoms for fixed cardinality data and string for variable cardinality.

I didn't expect that user code would typically access the internal fields of the Cldr.LanguageTag.t struct so if there is a way to do a better job to improve compatibility with your existing code then I'm open to ideas.

Please feel free to open a separate issue so we can discuss in more detail.

[setop]

That's very nice of you but you don't owe me any apology. Managnin dependencies is part of the job.

I'm maintaing a former branch of an application for which the main branch has already the code to handle the change of type. I'm good with the fix I found.

If you want more details : commit 951508d (25/01/22 05:50:37 +0800), between 2.25 and 2.26 introduced the change that breaks our code.

in lib/cldr/config/config.ex#L980
-  def known_number_systems_like(locale_name, number_system, config) do
+  def known_number_systems_like(locale_name, number_system, config) when is_atom(locale_name) do

The patch we made to fix it is there, (Mar 21, 2022 5:50pm GMT+0100).

Cheers