Sketch for UTF-16 support

[lemire]

This cannot be merged because we have no testing for UTF-16.

The proposed strategy here is to convert everything in UTF-8. Note that our entire codebase so far is UTF-8 specific: it makes no sense to accept anything but UTF-8 unless we do a fair amount of reengineering.

It is common these days to have the core engine run in UTF-8 and to view UTF-16 as something you convert from or to.

[anonrig]

This code is already included in the main branch (with your latest commit)

[anonrig]

From my understanding, encoding is only supported in URL spec due to the percent_encode function that takes encoding as a parameter. I tried looking for test fixtures provided by the URL spec that focuses on any other input than UTF-8.

What is ambiguous is that percent-encode after encoding, with encoding (parameter) is only used inside query state. Let us keep the encoding parameter as it is and add encoding support to percent_encode preferably as a parameter.

[lemire]

@anonrig It should be documented that the input is always expected to be UTF-8 and that the encoding parameter refers to something else.

[lemire]

Closing.

[mertcanaltin]

greetings I want to work on this subject