Preserve BOM in web editor

[silverwind]

The ToUTF8* functions were stripping BOM, while BOM is actually valid in UTF8, so the stripping must be optional depending on use case. This does:

• Add a options struct to all ToUTF8* functions, that by default will strip BOM to preserve existing behaviour
• Remove ToUTF8 function, it was dead code
• Rename ToUTF8WithErr to ToUTF8
• Preserve BOM in Monaco Editor
• Remove a unnecessary newline in the textarea value. Browsers did ignore it, it seems but it's better not to rely on this behaviour.

Fixes: #28743
Related: #6716 which seems to have once introduced a mechanism that strips and re-adds the BOM, but from what I can tell, this mechanism was removed at some point after that PR.

[wxiaoguang]

I think this fix is not complete, and it is also related to #28119 (web editor EOL) and others, they are the sample problem.

The key problem is not "what MonacoEditor does", but it is "what Gitea should do".

Think about some files in a git repo:

• A file without BOM
• A file with UTF-8 BOM
• A file with UTF-16 BOM
• A file with "\n"
• A file with "\r\n"

By existing approach, they could all be edited by MonacoEditor. The real problem is when "saving" the files, backend code does incorrect assumption: it only recognizes UTF-8 or "\n". So UTF-16 BOM and "\r\n" files are all changed unexpectedly.

So, the complete approach could be like this IMO: when saving, the backend code should first detect the file kind (BOM, EOL), then convert the submitted text to the expected format (correct BOM, correct EOL), then everything should work well.

[silverwind]

So, the complete approach could be like this IMO: when saving, the backend code should first detect the file kind (BOM, EOL), then convert the submitted text to the expected format (correct BOM, correct EOL), then everything should work well.

I'm not sure such detection is neccesary. I would just ensure the file round trips through backend and frontend unmodified while editing.

EOL is a more complicated topic because of the possibility for mixed EOL, but BOM is imho quite a clearcut case: If it's there, keep it. Ideally even indicate its precence in the editor.

So for BOM, we should render it without BOM for display (see #6716 regarding rendering artifacts). Only when passing to the editor, preserve it and this is exactly what this PR does. I will test if UTF16 BOM works as well with this PR, I assume it will.

[silverwind]

It seems UTF-16 rendering is broken in Gitea and a number of other tools. See this file. Gitea, VSCode, Monaco and Sublime Text all seem to convert it to UTF-8 and they render this:

GitHub renders the letters "test" correctly but shows a broken BOM rendering:

So I think UTF-16 is a diffferent topic for another PR. We should first render it correctly, then we can talk about BOM there.

[wxiaoguang]

Maybe there could be some improvements, eg: Remove ToUTF8WithFallback , and introduce "AutoFallback" option for "ConvertOpts"). And ToUTF8DropErrors looks strange, ideally there could be an option like IgnoreBroken.

And as the comment above, to resolve the UTF-16 BOM and "\r\n" problems, it needs a new approach.

Anyway, they are all legacy problems.

[silverwind]

Merging as-is. More improvements can come later. I agree we should move all function variants to the options struct.