Support umlauts (ö ä ü) in the domain name

[metmarkosaric]

Support umlauts (ö ä ü) in the domain name

request via twitter

Update: Umlauts and other special characters (including numbers, dots and slashes) are now supported so you can add international domain names to Plausible too.

[masteradhoc]

+1 request for this issue

[tobiasgr]

+1 from here, but with æ,ø,å ;)... what I do in my own system is to convert to punycode in the setter, and convert back from punycode in the getter. And all matches are made with the punycode version of a domain...

[vinibrsl]

An easy solution would be to transform all domains to Punycode on creation. This wouldn't require any data migration, as plausible.io is plausible.io in Punycode as well. But èé.fr, for instance, is xn--8cac.fr in Punycode. This can be done using a library we already have: :idna.encode/2.

This enables support for International Domain Names (IDNs), but two UX issues may arise.

1. IDNs would be displayed as Punycode (xn--) unless all client-side + e-mail references are changed

     <h2 class="text-xl font-bold dark:text-gray-100">Waiting for first pageview</h2>
-    <h2 class="text-xl font-bold dark:text-gray-100">on <%= @site.domain %></h2>
+    <h2 class="text-xl font-bold dark:text-gray-100">on <%= Plausible.Site.IDN.decode(@site.domain) %></h2>

2. URLs would use Punycode, e.g. https://plausible.io/xn--80andih6b.com

What are your thoughts on this, @ukutaht?

[ukutaht]

An easy solution would be to transform all domains to Punycode on creation

Hmm I don't understand what problem this solves. I would think that the browser, HTTP, Elixir, Postgres and Clickhouse all support UTF-8 as a character encoding, which already supports special characters. Before we can evaluate Punycode as a solution, we must first consider what problem we're actually solving. Are any of the underlying technologies limited to ASCII?

[ukutaht]

If not, then what is limiting us from creating domains with special characters at the moment?

[masteradhoc]

@ukutaht Currently your not allowing us to add this.
Example: äö-test.li

As @vinibrsl wrote we have to currently manually translate it punycode compatible to: xn---test-fra0k.li
then it works :)

But its not readable at all and misleading to our clients.

[ukutaht]

Yes I agree that it's currently not allowed in the UI, that's a problem, and we want to fix it.

I wanted to continue the conversation about the technical implementation that @vinibrsl proposed. The recommendation was to encode to Punycode when storing the domain and to decode it when pulling it out of the database. My question is: why is Punycode needed here? Can't we just store the domain as utf-8 which already supports all special characters you might want to use in a domain name?

[kthy]

Storing as Punycode makes it easier for you to tell the difference between аlpha.com (xn--lpha-43d.com) and alpha.com (alpha.com) internally.

[vinibrsl]

Hey everyone! #2034 adding international domain names support has been merged 🙌

It adds support for any Unicode character in the "letter" category. In other words, a letter in any language. Plus numbers, dots, and slashes.

[metmarkosaric]

nice one @vinibrsl! so it's live already now? i still get the same error

[vinibrsl]

I'll deploy it today. Should be live in an hour :) @metmarkosaric

[metmarkosaric]

cool! i'll prepare the announcement

[tobiasgr]

💪