Recommend/convert to use case-sensitive collation for MySQL/MSSQL

[wxiaoguang]

Mainly for MySQL/MSSQL.

It is important for Gitea to use case-sensitive database charset collation. If the database is using a case-insensitive collation, Gitea will show startup error/warning messages, and show the errors/warnings on the admin panel's Self-Check page.

Make gitea doctor convert work for MySQL to convert the collations of database & tables & columns.

• Fix MySQL/MariaDB: Unique Constraint on Branches doesn't respect casing: UQE_branch_s #28131

⚠️ BREAKING ⚠️

It is not quite breaking, but it's highly recommended to convert the database&table&column to a consistent and case-sensitive collation.

[wxiaoguang]

The problem

In history, Gitea doesn't strictly require charset/collation. It causes a lot of problems. Especially for MySQL/MSSQL:

• Emoji causes errors (MySQL/MariaDB utf8mb3).
• UNIQUE conflicts (branch name main vs Main) if the collation is case-insensitive.
• SQL error when trying to compare columns with different collations.
• Ordering problem, different collation generates different "ORDER BY" result.
• Searching problem. LIKE '%...%' behaves differently for case-sensitive and case-insensitive collations.
• MySQL/MariaDB have different collation systems but Gitea only have "MySQL" database type.

The solution

So maybe it's the time to fix the problem.

Ideally, Gitea should require strict and consistent collations. Considering there are a lot of old instances which may already have various database collations, this PR only does some simple fix and problem detection, and don't really force users to do anything.

This PR does:

1. Only improve for MySQL/MSSQL. These two are mostly affected by the problems.
   • MSSQL has various limitations, so there is no "converting" solution for it at the moment.
2. Introduce a config option [database].CHARSET_COLLATION
   • If it is empty, Gitea will try to find a default collation. Try to use utf8mb4_0900_as_cs/uca1400_as_cs for MySQL/MariaDB, and fallback to utf8mb4_bin if the database is too old. Try to use Latin1_General_CS_AS for MSSQL.
   • If it is not empty, for example, end users would really like to customize their collation, Gitea could also accept it, even if it is case-insensitive. Then the old behaviors could be kept (not too much breaking)
3. Introduce a concept ExpectedCollation: [database].CHARSET_COLLATION if not emtpy, or the one found by Gitea (above).
4. When Gitea starts, it tries to alter the empty database's collation to the ExpectedCollation (quite fast)
5. Gitea will check database collations and all column collations, to find "unexpected collations"
   • These "unexpected collations" will be reported as startup error/warning logs and will be shown on admin panel's Self Check page.
6. If end users see the error/warning:
   1. They could choose to do nothing, just keep using the "unexpected collations". Well, it would cause problems in rare cases, indeed that's the user's choice.
   2. Run gitea doctor convert, then the database and columns will be converted to ExpectedCollation.
      • Only MySQL has such support at the moment.
      • MSSQL users need to do the converting manually.

---

The admin panel self-check page also partially addresses #27011

Then after gitea doctor convert:

[wxiaoguang]

Found another bug for MSSQL testing: Make SQL work with case-sensitivie SQLServer go-testfixtures/testfixtures#182

---

I think it doesn't need to wait for the go-testfixtures fix. This PR doesn't affect MSSQL at the moment, so the MSSQL fix could be done later separately, the approach is general enough to support it.

[lunny]

One review needed and Last Call @go-gitea/technical-oversight-committee