Add explicit-endian String::from_utf16 variants

[CAD97]

This adds the following APIs under feature(str_from_utf16_endian):

impl String {
    pub fn from_utf16le(v: &[u8]) -> Result<String, FromUtf16Error>;
    pub fn from_utf16le_lossy(v: &[u8]) -> String;
    pub fn from_utf16be(v: &[u8]) -> Result<String, FromUtf16Error>;
    pub fn from_utf16be_lossy(v: &[u8]) -> String;
}

These are versions of String::from_utf16 that explicitly take UTF-16LE and UTF-16BE. Notably, we can do better than just the obvious decode_utf16(v.array_chunks::<2>().copied().map(u16::from_le_bytes)).collect() in that:

• We handle the case where the byte slice is not an even number of bytes, and
• In the case that the UTF-16 is native endian and the slice is aligned, we can forward to String::from_utf16.

If the Unicode Consortium actively defines how to handle character replacement when decoding a UTF-16 bytestream with a trailing odd byte, I was unable to find reference. However, the behavior implemented here is fairly self-evidently correct: replace the single errant byte with the replacement character.

[rust-highfive]

Thank you for submitting a new PR for the library teams! If this PR contains a stabilization of a library feature that has not already completed FCP in its tracking issue, introduces new or changes existing unstable library APIs, or changes our public documentation in ways that create new stability guarantees then please comment with r? rust-lang/libs-api @rustbot label +T-libs-api to request review from a libs-api team reviewer. If you're unsure where your change falls no worries, just leave it as is and the reviewer will take a look and make a decision to forward on if necessary

[rust-highfive]

r? @joshtriplett

(rust-highfive has picked a reviewer for you, use r? to override)

[ChrisDenton]

Surely it would be more consistent with String::from_utf16 and u16::from_be to take a &[u16] slice? This would also sidestep the trailing byte issue.

[CAD97]

The point of this API surface is to take u8 units like u16::from_be_bytes. If you read a file, you get Vec<u8>, so in order to read a UTF-16 file, someone needs to decode the &[u8] into UTF-16. For the case where you already have &[u16], we have char::decode_utf16.

[ChrisDenton]

Ah, I see, it's to deserialize bytes from say a file or other stream. That makes sense.

[CAD97]

(triaging my open PRs)

I've inlined this into the project that needed it, and it's been working fine there. (Though tbf I haven't exercised that codepath much.)

I think this is right on the edge of interesting enough to include in std; the forwarding to the aligned version when possible is beneficial but easy to overlook.

The one thing that maybe should be added is a note to use decode_utf16 if you already have native-endian &[u16], to avoid a &[16] -> &[u8] -> &[u16] loop just to call these functions (and to explain the lack of a ne variant). Along those lines, maybe these should be named with _bytes in the name to match the convention from u16 where [to|from]_xe works on just u16 but [to|from]_xe_bytes converts to [u8; 2].

I'm happy to do either, just S-waiting-on-review.

[Kijewski]

Maybe it would be better to use String::with_capacity() + String::push() instead of collect::<Result<_, _>>() until #48994 is fixed?

[CAD97]

Doing this would match the implementation in from_utf16. However, I expect these functions will be much less performance critical, so adding the extra code is perhaps unneeded? I'm not sure.

[pitaj]

Ping from triage @joshtriplett. It's been 10 months since last activity.

[dtolnay]

This looks great to me. Please open a tracking issue.

maybe these should be named with _bytes in the name to match the convention from u16

I find the names currently in the PR to be all right and would be happy landing them as is. I'm sure we'll discuss the naming further as part of stabilization, though.

Thanks!

[CAD97]

Tracking issue linked: #116258

[dtolnay]

Thank you!

[dtolnay]

@bors r+

[bors]

📌 Commit 5facc32 has been approved by dtolnay

It is now in the queue for this repository.