Remove TranslateUnicodeToOem and all related code #14745
Conversation
lhecker
added
Product-Conhost
src/inc/til/type_traits.h
Outdated
| }; | ||
| // std::span remains largely unused in this code base so far. | ||
| //template<typename U, std::size_t E> | ||
| //struct is_contiguous_view<std::span<U, E>> : std::true_type |
There was a problem hiding this comment.
IMO no reason to have it commented out, it'll just be one more place to change when we do use std::span. uncomment?
There was a problem hiding this comment.
We don't import the <span> anywhere yet, so the type usually isn't defined. I'm planning to simply replace all gsl::span usage with std::span in the future. Matter of fact, I might just be doing it right now.
There was a problem hiding this comment.
#14763 just merged so you're good to go now haha
|
|
||
| // TODO: This is also rather strange and will also probably make more sense if we stop guessing that we need 2x buffer to convert. | ||
| // This might need to go on the other side of the fence (inside host) because the server doesn't know what we're going to do with initial num bytes. |
There was a problem hiding this comment.
Do we still need to figure out whether this "might"?
There was a problem hiding this comment.
To be entirely honest I don't quite get what this comment means and what InitialNumBytes is supposed to do. I can try to figure it out, but it can probably also be left as is for the time being.
|
Also related to #4551? |
ghost
added
Area-CookedRead
|
Also fixes #7589. |
When working on #14745 I found `KeyEvent`s a little hard to read in the debugger. I noticed that this is because of sign extension when converting `char`s to `wchar_t`s in `KeyEvent::SetCharData`.
When working on #14745 I noticed that `SplitToOem` was in a bit of a poor state as well. Instead of simply iterating over its `deque` argument and writing the results into a new `deque` it used `pop` to advance the head of both queues. This isn't quite exception safe and rather bloaty. Additionally there's no need to call `WideCharToMultiByte` twice on each character if we know that the most verbose encoding is UTF-8 which can't be any more than 4 chars anyways. Related to #8000. ## PR Checklist * 2 unit tests cover this ✅
|
Now after rewriting half of Also:
|
lhecker
changed the title
microsoft-github-policy-service
bot
added
Priority-1
| struct is_contiguous_view<std::basic_string_view<U, V>> : std::true_type | ||
| { | ||
| }; | ||
| #ifdef GSL_SPAN_H | ||
| template<typename U, std::size_t E> | ||
| struct is_contiguous_view<std::span<U, E>> : std::true_type |
There was a problem hiding this comment.
Ah woops. 😅 Well, problem fixed!
#14745 contains two regressions related to console alias handling: * When `ProcessAliases` expands the backup buffer into (an) aliased command(s) it changes the `_bytesRead` field of `COOKED_READ_DATA`, requiring us to re-read it and reconstruct the `input` string-view. * Multiline aliases are read line-by-line whereas #14745 didn't treat them any different from regular single-line inputs. ## Validation Steps Performed In `cmd.exe` run ``` doskey test=echo foo$Techo bar$Techo baz test ``` The output should look exactly like this: ``` C:\>doskey test=echo foo$Techo bar$Techo baz C:\>test foo C:\>bar C:\>baz C:\> ```
|
Just wanted to check: With this PR merged, is the |
|
Yes, the test case as described in #4551 is now fixed. However, I believe that "improved" is just the right choice of words: There's still no consistent internal support for surrogate pairs for instance, but this is something that I'm actively working on. |
|
Has this fix shipped in Windows 11 conhost already? |
|
Allegedly this will not ship for quite a while unfortunately. 10000s of lines of other changes are also waiting to be shipped, including massive correctness and performance improvements, and I'm quite excited to see when it does. |
|
(It will ship in the Terminal in 1.18 SoonTM though) |
|
Thank you! |
|
Is this related to #12626? |
The overarching intention of this PR is to improve our Unicode support. Most
of our APIs still don't support anything beyond UCS-2 and DBCS sequences.
This commit doesn't fix the UTF-16 support (by supporting surrogate pairs),
but it does improve support for UTF-8 by allowing longer
charsequences.It does so by removing
TranslateUnicodeToOemwhich seems to have had an almostviral effect on code quality wherever it was used. It made the assumption that
all narrow glyphs encode to 1
charand most wide glyphs to 2chars.It also didn't bother to check whether
WideCharToMultiBytefailed or returneda different amount of
chars. So up until now it was easily possible to readuninitialized stack memory from conhost. Any code that used this function was
forced to do the same "measurement" of narrow/wide glyphs, because of course
it didn't had any way to indicate to the caller how much memory it needs to
store the result. Instead all callers were forced to sorta replicate how it
worked to calculate the required storage ahead of time.
Unsurprisingly, none of the callers used the same algorithm...
Without it the code is much leaner and easier to understand now. The best
example is
COOKED_READ_DATA::_handlePostCharInputLoopwhich used to contain 3blocks of almost identical code, but with ever so subtle differences. After
reading the old code for hours I still don't know if they were relevant or not.
It used to be 200 lines of code lacking any documentation and it's now 50 lines
with descriptive function names. I hope this doesn't break anything, but to
be honest I can't imagine anyone having relied on this mess in the first place.
I needed some helpers to handle byte slices (
std::span<char>), which is whya new
til/bytes.hheader was added. Initially I wrote abuf_writerclassbut felt like such a wrapper around a slice/span was annoying to use.
As such I've opted for freestanding functions which take slices as mutable
references and "advance" them (offset the start) whenever they're read from
or written to. I'm not particularly happy with the design but they do the job.
Related to #8000
Fixes #4551
Fixes #7589
Fixes #8663
Validation Steps Performed