WINDOWS: make env.jl and file.jl unicode-compliant for #4240

[vtjnash]

No description provided.

[vtjnash]

@JeffBezanson is this acceptable?

[JeffBezanson]

No; sizeof(void*) is big enough to hold an Int64 or Float64, so this amounts to over-allocating essentially every array. Due to alignment this will actually add 16 bytes, which is a lot of overhead if you have lots of small arrays. The size of our array metadata (64 bytes) is already killer for lots of small objects.

[vtjnash]

Please suggest another approach then

Perhaps ccall could push! a hidden zero on every array, and set a flag bit?

[JeffBezanson]

I don't know; this is a very hard problem. Calling push! internally is a non-starter; it's very expensive, and ccall might not always have access to the array object itself. Maybe the UTF16String constructor can handle it?

[JeffBezanson]

Or adding a 0 if elsz <= 2?

[Keno]

Why exactly do we need this?

[vtjnash]

For strlen

If utf16string is going to handle this in Julia, then perhaps everything should. This really just feels too magical, and therefore problematic, since the user has so many avenues for making strings.

[JeffBezanson]

We have to think in detail about what cases we can and can't handle. For example, if you allocate data in C and wrap it with pointer_to_array, there is nothing we can do short of copying it. The hack we use now is effective because passing julia-allocated byte arrays to C is extremely common. Without the if elsz==1 hack we would be totally hosed. The existence of ways to work around it does not matter; what matters is the 99% cases.

[stevengj]

It seems reasonable to me to handle this in UTF16String, since the most important reason for that type is for Windows API calls.