Strings RFC #1848

rylev · 2022-06-24T12:17:17Z

This is an RFC for how we'll handle the many types of Strings in Rust and Windows and how they interact with each other.

This is a complicated question, so I imagine we'll probably not get it right, but I do believe this is an improvement over the status quo not only in terms of useability but also safety.

mbartlett21 · 2022-06-26T23:18:49Z

rfcs/003-strings.md

+    fn as_wide(&self) -> &[u16] {}
+    // String data with the trailing 0
+    fn as_wide_with_nul(&self) -> &[u16] {}
+    fn to_string(&self) -> Result<String, FromUtf16Error> {}


Note: This function collides with ToString::to_string, since HSTRING implements Display below.

Perhaps this could be renamed try_to_string?

Good catch. That's a shame. We could use into_string which mirrors CString::into_string directly. I originally didn't do this as it's not exactly trivial to reuse the String's buffer for the HSTRING buffer and thus I didn't want to take self.

mbartlett21 · 2022-06-26T23:24:20Z

rfcs/003-strings.md

+
+/// The wide equivalent to std::ffi::CString;
+#[repr(transparent)]
+struct CWString(..);


Should there be a matching borrowed CWStr, like CString and CStr?

Perhaps... I'm not sure how often this will be useful in practice though it might be. PCWSTR would just be like a raw pointer so ideally the user would only rarely use it, but I'm unsure how often you'd want a &CWStr and not a &CWString

@rylev
One reason would be so there isn't a double-indirection.
It can also be used to convert a PCWSTR into a safely-used reference without having to copy or own the data. (See for example CStr::from_ptr)

mbartlett21 · 2022-06-26T23:56:01Z

rfcs/003-strings.md

+
+These function build null terminated string data of the appropriate width (`u8` in the case of `PCSTR` and `u16` in the other cases).
+
+**question**: for all these types built from static strings, could we special case them so that they don't use reference counting at all?


You could just special case for a string literal, then use that with concat!($str, "\0"), etc. For wide strings, you'd probably need something like const_heap (nightly api very far from stable) or compiler support.

This refers to whether we need a special version of HSTRING and the like that doesn't track reference counts since its backing memory is statically allocated. Actually statically allocating enough memory for a wide string is already a solved problem (e.g., see here).

Ok. I didn't realise that the const on stable had got that far. EDIT: And I forgot that you could use it to get a length for a constant array...

So essentially, you'd get a wide string reference (PCWSTR), then pass that to WindowsCreateStringReference (in windows::Win32::System::WinRT), then use the resultant HSTRING structure.

Using WindowsCreateStringReference is generally a pessimization unless the calling language natively supports null terminated wide string literals, like C# and C++. For languages like Rust, that's not worth it as the resultant HSTRING typically ends up being copied almost immediately.

kennykerr · 2022-06-27T13:33:04Z

rfcs/003-strings.md

+
+It's clear that "IN" params are the scenario where ownership differs from the status quo. Therefore, the `windows` crate will treat the types as the following.
+
+* Introduce a new type `CWString` which is the wide string equivalent to `CString`.


I'll review more carefully when I get a moment but my most obvious question is what CWString gives you that HSTRING doesn't already provide. Obviously, there are already a dizzying number of string types and I'd love to not make that dizzier. 😉

Maybe? The difference is that HSTRING has the added complexity of being reference counted. CWString would have the added benefit of being symmetrical with the Rust std library CString type, but I agree that we want to try to limit the explosion of types if we can...

kennykerr · 2022-06-28T20:02:45Z

rfcs/003-strings.md

+
+#### Windows
+
+Windows has the following string types:


Another way to look at this is that Windows has a variety of string types that need to be supported for interop with various APIs but that languages can and should only use them on the ABI boundaries as much as is practical. The extreme example of this is C# where only System.String exists in C# and all the different string types are marshaled away behind the scenes. While I'm not suggesting we go quite that far, we should try to think of these as interop types needed only for the ABI boundary and put our energy into making it as efficient and seamless as possible to go from and get back to native and idiomatic Rust string types, and not try to make Windows replacements with all the bells and whistles of Rust's string types.

For PCSTR, we can just use *const c_char directly, and CStr provides from_ptr and as_ptr methods to convert to and from it. For PSTR, we would need CString and CStr to allow mutability.

For PWSTR and PCWSTR, it would need roughly the same, except with wide characters. PCWSTR (I think) should be able to be created from a BSTR or HSTRING.

That would be the ideal situation.

However, the unfortunate situation with Rust is, that Rust does not provide a zero-overhead string type that's compatible with Windows' native string encoding. Vec<u16>/&[u16] is the best Rust has to offer in this respect. I don't see any way "to go from and get back to native and idiomatic Rust string types" at all.

Being pragmatic here, and assuming that unmatched surrogate pairs won't appear in practice is a dangerous route to go down. Doing so will open up code based on the windows crate to attacks, even if only DoS attacks.

Indeed that's the ideal hence the "as much as is practical". So we need a null terminated wide char but we need such things as little as humanly possible with the emphasis on interop rather than string features. And since we need HSTRING anyway and HSTRING is a null terminated wide char string that should cover those needs. Even if we end up calling it CWString, for symmetry, that just happens to be implemented as an HSTRING we can at least minimize the number of implementations we're carrying around.

That makes sense. I am a bit skeptical that we'll actually save much code by using an HSTRING as a backing implementation detail, but if this is truly just an implementation detail then it doesn't really matter. We can keep playing with the implementation without impacting users.

mbartlett21 · 2022-06-28T23:44:56Z

rfcs/003-strings.md

+
+* `PSTR`: a mutable, nul-terminated string (i.e., composed of "characters" a.k.a. `u8`s) that is often but not always ANSI encoded.
+* `PCSTR`: an immutable version of `PSTR`.
+  * **QUESTION**: what are the practical differences between this and `PSTR`?


The difference is that PCSTR can be created from a CStr reference, and can also be converted back.
EDIT: That is because CStr doesn't allow mutability.

ChrisDenton · 2022-07-04T16:56:19Z

I would also add that there are some more rough edges with functions that take even more variations on strings.

There's UNICODE_STRING which is roughly equivalent to the Rust types &[u16], &mut [u16] or Vec<u16> depending on the function being used and the value of MaximumLength. This can support interior nulls.

Also there are functions such as CompareStringOrdinal that can take a null terminated LPCWCH string but also optionally take a length so the strings need not be null terminated (and may therefore include nulls).

And some functions such as SetFileInformationByHandle that take structures which require both a null terminated string and the length of the string.

While these should not be too difficult to support, I think it's worth noting there are a number of variations on the basic ptr + length string.

kennykerr · 2022-07-06T17:30:14Z

Right, which is why I'm reluctant to add some new synthetic string type like CWString when we already have so many to contend with. If anything, something like CWString belongs in the standard library alongside CString while the windows crate should focus on those that are specific to Windows, like HSTRING and BSTR, as it does.

To @ChrisDenton's point, it is also not the goal of the window crate to smooth over every last detail of string handling in the Windows API. There will always be APIs that are harder to call directly and possibly deserve some specialized crate to provide a more idiomatic experience if the API is that important. Many are not. 😏

kennykerr · 2022-07-06T18:08:39Z

rfcs/003-strings.md

+```rust
+let x: CWString = w!("hello");
+let y: CString = c!("hello");
+let z: HSTRING = h!("hello");


I don't think we need these macros to produce allocated strings. That requires runtime support. They should all be able to return const string literals of PCWSTR and PCSTR respectively. Then they can simply be used to call functions like MessageBoxA/W directly and efficiently.

Common Windows string types like HSTRING and BSTR can then just provide From<&T> implementations for PCWSTR for cases where a string needs to be computed at runtime and in future if/when CString and CWString are available in the std library we can just provide From implementations for those as well.

Then we shouldn't need the Into<T> below, unless I'm forgetting something.

Here's what I have in mind: https://github.com/microsoft/windows-rs/compare/utf16?expand=1

And this is compatible with the existing state of the windows crate so would make it easier to land the Borrowed PR.

This proposal is a subset of what I had in mind so I think it's fine to try it out. We can always expand from there to have dedicated CWString types and such.

riverar · 2022-07-08T21:19:48Z

rfcs/003-strings.md

+* `PSTR`: a mutable, nul-terminated string (i.e., composed of "characters" a.k.a. `u8`s) that is often but not always ANSI encoded.
+* `PCSTR`: an immutable version of `PSTR`.
+  * **QUESTION**: what are the practical differences between this and `PSTR`?
+* `PWSTR`: a mutable, nul-terminated "wide" string (i.e., composed of "wide characters" a.k.a. `u16`s) that is often but not always UTF-16 encoded.


[...] that is often but not always UTF-16 encoded.

Can you expand on this? I'm not aware of any non-UTF-16 encoded PWSTRs. Are you instead trying to convey that a grapheme may span 1 or 2 code 16-bit units (2-4 bytes)?

It's possible to create invalid UTF-16 strings because the kernel does not check validity. This is exceedingly rare in practice but it is possible, especially if there's a malicious app causing mischief.

The prime candidate here is the filesystem. NTFS allows names to be just about any sequence of 16-bit values (with a few reserved code units). Those names will inevitably be returned from FindFirstFile/FindNextFile, so PWSTR/PWCSTR cannot make any assumptions about the validity of the UTF-16 encoding.

Likewise, named kernel objects (events, mutexes, pipes, ...) can have names that aren't valid UTF-16.

Not quite the norm, but invalid UTF-16 sequences are observed in the wild, usually as a result of an accident/bug, or, more frequently, in attacks.

There's no use trying to define what these are. They're just pointer typedefs, which is why windows-rs defines them as simply as possible. We can provide safe ways to create them and pass them to API calls, but we cannot make any guarantees about them or provide generally safe conversions from them.

kennykerr · 2022-08-23T16:47:52Z

This PR's pretty old. Can we move it along or just close it for now?

First draft of the strings RFC

1ac7cf7

rylev mentioned this pull request Jun 24, 2022

PROCESSENTRY32 szExeFile usage #1845

Closed

mbartlett21 reviewed Jun 26, 2022

View reviewed changes

kennykerr reviewed Jun 27, 2022

View reviewed changes

kennykerr reviewed Jun 28, 2022

View reviewed changes

mbartlett21 reviewed Jun 28, 2022

View reviewed changes

kennykerr reviewed Jul 6, 2022

View reviewed changes

riverar reviewed Jul 8, 2022

View reviewed changes

kennykerr closed this Aug 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strings RFC #1848

Strings RFC #1848

rylev commented Jun 24, 2022

mbartlett21 Jun 26, 2022

rylev Jun 27, 2022

mbartlett21 Jun 26, 2022

rylev Jun 27, 2022

mbartlett21 Jun 27, 2022

mbartlett21 Jun 26, 2022

rylev Jun 27, 2022

mbartlett21 Jun 27, 2022 •

edited

Loading

mbartlett21 Jun 29, 2022

kennykerr Jun 30, 2022

kennykerr Jun 27, 2022

rylev Jun 30, 2022

kennykerr Jun 28, 2022

mbartlett21 Jun 28, 2022

tim-weis Jul 1, 2022

kennykerr Jul 1, 2022

rylev Jul 4, 2022

mbartlett21 Jun 28, 2022 •

edited

Loading

ChrisDenton commented Jul 4, 2022

kennykerr commented Jul 6, 2022

kennykerr Jul 6, 2022

kennykerr Jul 6, 2022

kennykerr Jul 6, 2022

kennykerr Jul 8, 2022

kennykerr Jul 8, 2022

rylev Jul 11, 2022

riverar Jul 8, 2022

ChrisDenton Jul 9, 2022

tim-weis Jul 9, 2022

kennykerr Jul 9, 2022

kennykerr commented Aug 23, 2022


		These function build null terminated string data of the appropriate width (`u8` in the case of `PCSTR` and `u16` in the other cases).

		question: for all these types built from static strings, could we special case them so that they don't use reference counting at all?


		It's clear that "IN" params are the scenario where ownership differs from the status quo. Therefore, the `windows` crate will treat the types as the following.

		* Introduce a new type `CWString` which is the wide string equivalent to `CString`.

Strings RFC #1848

Strings RFC #1848

Conversation

rylev commented Jun 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbartlett21 Jun 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbartlett21 Jun 28, 2022 • edited Loading

Choose a reason for hiding this comment

ChrisDenton commented Jul 4, 2022

kennykerr commented Jul 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kennykerr commented Aug 23, 2022

mbartlett21 Jun 27, 2022 •

edited

Loading

mbartlett21 Jun 28, 2022 •

edited

Loading