Support Japanese charsets

[dhow]

Summary

I noticed that when AUTO_RESOLVE_TITLES is true I get th[1] error when I create short links from Japanese pages that use Japanese specific encoding such as shift_jis. I noticed that this doesn't happen on UTF-8 pages. It would be great if we can support Japanese encoding, in specific, the two Japanese traditional encodings[2].

Version: 4.1.0 / docker image: shlinkio/shlink@sha256:6d0fb5fb169bf76940d2c7c628aac78ea2dfbbd37e897c456c1ff198a6185f64
DB: MariaDB 11.3.2-MariaDB-1:11.3.2 / docker image: sha256:f0a6faee9d0e55492f238d1d11ff13d77616ea12d8c38bedf090da2ee05532be

[1] Error

$ shlink short-url:create "https://www.sony.jp/ichigan/products/SEL85F18/"

In ExceptionConverter.php line 90:
  An exception occurred while executing a query: SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect string value: '\x83f\x83W\x83^...' f or column `shlink`.`short_urls`.`title` at row 1

In Exception.php line 28:
  SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect string value: '\x83f\x83W\x83^...' for column `shlink`.`short_urls`.`title` at row 1

In Statement.php line 55:
  SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect string value: '\x83f\x83W\x83^...' for column `shlink`.`short_urls`.`title` at row 1

[2] Two most widely used Japanese traditional encoding

• EUC-JP
• Shift_JIS

Use case

Many users who create links from pages using Japanese encoding will benefit from this.

[dhow]

P.S.: I was unable to trace where the logic happens when auto_resolve_titles is true, I could try to implement this if someone could show me the file / line of the fetching title code. :)

[acelaya]

Yep, I can reproduce this, but this is more a bug than a feature request, as it should work.

The error is pretty strange. It's like it's trying to parse the vale as a date somehow? 😅

I'll investigate.

[dhow]

Hello @acelaya -- you might already find the solution but I have feeling it's related to this answer.

[acelaya]

What I can confirm is that, if you manually pass the title with the same value that's set in https://www.sony.jp/ichigan/products/SEL85F18/ title, then it works, so the problem is happening when "reading" the page.

[acelaya]

Yep, the title needs to be re-encoded. If I apply mb_convert_encoding($title, 'utf8', 'shift_jis') it works, so I guess the fix would require reading the <meta charset="??" /> from the page and using it.

[dhow]

Yes @acelaya that's the same way I'd approach this problem -- reading charset in header and use that to feed mb_ functions is the way to go. If you manage to fix it, please share your commits, I'd like to learn. Thanks!!

[acelaya]

#2135 should do the trick.

[acelaya]

I have just released Shlink 4.1.1 which includes this fix.

[dhow]

I just confirmed this in 4.1.1 docker -- working perfectly! Thanks so much @acelaya !