Ninja UTF8 conversion: Using native windows API for conversion. #1671

jlonnberg · 2019-11-08T14:24:15Z

WIP for issue #1195.
Updated version using windows native API for converting between wide and utf8 for increased speed. Fixed buildissue for mingw and added needed defines to the cmake script.

Metrics for building chromium
`
Ninja Master
metric count avg (us) total (ms)
.ninja parse 4849 1931.3 9364.8
canonicalize str 1889079 0.0 58.5
canonicalize path 1985429 0.0 42.8
lookup node 1932008 0.0 36.7
.ninja_log load 1 1504.0 1.5
.ninja_deps load 1 1314.0 1.3
node stat 7363 285.0 2098.4
depfile load 22 184.6 4.1
StartEdge 823 5840.9 4807.1
FinishCommand 822 3970.5 3263.8
CLParser::Parse 753 618.3 465.6

Ninja UTF8-version
metric count avg (us) total (ms)
.ninja parse 4849 2104.7 10205.6
canonicalize str 1889079 0.0 90.5
canonicalize path 1985429 0.0 60.8
lookup node 1933885 0.1 169.9
.ninja_log load 1 2000.0 2.0
.ninja_log recompact 1 14878.0 14.9
node stat 8748 260.8 2281.1
.ninja_deps load 1 3960.0 4.0
depfile load 22 209.7 4.6
StartEdge 823 5909.6 4863.6
FinishCommand 822 908.7 746.9
CLParser::Parse 753 615.2 463.2
`

configure.py

src/clparser_perftest.cc

src/disk_interface.cc

src/includes_normalize-win32.cc

src/util.h

jhasse · 2019-11-08T14:33:25Z

Sorry, missed the WIP, feel free to ignore my comments. Please add WIP to the title of the PR though.

jlonnberg · 2019-11-12T10:31:04Z

Should the final commit be pushed as a new pull-request, removing all the small formatting commits from the log?

jhasse · 2019-11-12T10:44:55Z

Not as a new PR. You can force push to your branch which will update the PR. I can also squash and merge via GitHub to get rid of the fix-up commits.

CMakeLists.txt

configure.py

src/clparser_perftest.cc

src/disk_interface.cc

src/includes_normalize-win32.cc

…to string::copy.

CMakeLists.txt

tristanlabelle

Awesome to see this! I'm looking forward to see if this resolves some of the bugs we've hit here on the Visual C++ team. I know there's a few more pieces that will be needed, but this change is a necessary first step for Ninja to work in international and mixed language environments and a great improvement. Thanks!

src/util.cc

src/subprocess-win32.cc

src/disk_interface.cc

src/clparser_perftest.cc

src/util.cc

jhasse · 2019-11-13T20:02:24Z

src/test.cc

-  if (!GetTempPath(sizeof(buf), buf))
+  TCHAR buf[MAX_PATH];
+  // GetTempPath returns zero if no path is found
+  if(GetTempPath(MAX_PATH, buf) == 0)
    return "";


Just as a reminder: Ninja doesn't use exceptions.

The empty return is the same as before, so it should not change any behavoiur.

src/test.cc

-  if (!GetTempPath(sizeof(buf), buf))
+  TCHAR buf[MAX_PATH];
+  // GetTempPath returns zero if no path is found
+  if(GetTempPath(MAX_PATH, buf) == 0)


src/test.cc

@@ -75,10 +75,12 @@ char* mkdtemp(char* name_template) {

 string GetSystemTempDir() {
 #ifdef _WIN32
-  char buf[1024];


tristanlabelle · 2020-02-21T21:19:01Z

@jhasse Are there still blockers for accepting and merging this?

tristanlabelle · 2020-03-17T20:59:16Z

Ping?

jlonnberg · 2020-03-18T06:45:54Z

@MrTrillian Fixed the merging conflict, branch is ready and building again.

tristanlabelle · 2020-03-30T16:21:25Z

@jhasse Can you provide an update on getting this PR merged? Any remaining steps? Thanks!

jhasse · 2020-03-30T16:36:31Z

I would like to see #1724 fixed first.

tristanlabelle · 2020-03-31T18:11:33Z

@jhasse That issue has been open for two months without activity, and it doesn't seem related? This PR has been ongoing for almost 5 months. How do we avoid this being prolonged indefinitely?

jhasse · 2020-04-01T07:33:02Z

By fixing #1724.

jhasse · 2020-04-01T10:37:40Z

I don't have time to look at this and I don't want to merge PRs without reviewing them. #1724 is next on my to-do list.

…ld#1671

tristanlabelle · 2020-08-14T18:40:11Z

@jhasse , #1724 is now closed, any chance you could look into this one?

jhasse

Had another look :)

Would be great if the commits could be cleaned up a little bit before merging.

jhasse · 2020-08-18T19:44:47Z

CMakeLists.txt

@@ -17,9 +17,12 @@ endif()
 # --- compiler flags
 if(MSVC)
 	set(CMAKE_MSVC_RUNTIME_LIBRARY "MultiThreaded$<$<CONFIG:Debug>:Debug>")
-	string(APPEND CMAKE_CXX_FLAGS " /W4 /GR- /Zc:__cplusplus")
+	string(APPEND CMAKE_CXX_FLAGS /W4 /GR- /Zc:__cplusplus -DUNICODE)


Why did you remove the quotes?

jhasse · 2020-08-18T19:46:15Z

CMakeLists.txt

@@ -30,6 +33,10 @@ else()
 	endif()
 endif()

+if(WIN32)
+	string(APPEND CMAKE_CXX_FLAGS " -DNOMINMAX")


Hm ... maybe target_compile_definitions instead?

Using target_compile_definitions (with PRIVATE) the definition won't be inherited by other applications who use ninja as a library.

Public header files shouldn't include windows.h at all.

jhasse · 2020-08-18T19:46:58Z

CMakeLists.txt

 else()
-	include(CheckCXXCompilerFlag)
+  if(WIN32)
+		string(APPEND CMAKE_CXX_FLAGS " -municode")


Would -DUNICODE work with MinGW, too? Then it could be moved to the if(WIN32) block below.

The -municode is needed to have mingw use the wmain entry point instead of the ordinary main.

jhasse · 2020-08-18T19:47:22Z

CMakeLists.txt

+  if(WIN32)
+		string(APPEND CMAKE_CXX_FLAGS " -municode")
+	endif()
+  include(CheckCXXCompilerFlag)


Please leave tab.

jhasse · 2020-08-18T19:47:54Z

configure.py

@@ -357,6 +359,9 @@ def binary(name):
        pass
    if platform.is_mingw():
        cflags += ['-D_WIN32_WINNT=0x0601', '-D__USE_MINGW_ANSI_STDIO=1']
+        cflags += ['-DNOMINMAX']


add to the line above.

jhasse · 2020-08-18T19:57:50Z

src/disk_interface.cc

-                             TimeStampFromFileTime(ffd.ftLastWriteTime)));
-  } while (FindNextFileA(find_handle, &ffd));
+    stamps->emplace(std::move(lowername),
+                             TimeStampFromFileTime(ffd.ftLastWriteTime));


remove spaces (or clang-format)

jhasse · 2020-08-18T20:03:19Z

src/subprocess-win32.cc

@@ -121,18 +122,20 @@ bool Subprocess::Start(SubprocessSet* set, const string& command) {
      CloseHandle(nul);
      pipe_ = NULL;
      // child_ is already NULL;
-      buf_ = "CreateProcess failed: The system cannot find the file "
+      buf_ =


no need to touch this line

jhasse · 2020-08-18T20:03:41Z

src/subprocess.h

@@ -53,7 +54,7 @@ struct Subprocess {

 private:
  Subprocess(bool use_console);
-  bool Start(struct SubprocessSet* set, const string& command);
+  bool Start(struct SubprocessSet* set, const StringPiece command);


remove const (from declaration)

jhasse · 2020-08-18T20:08:41Z

src/util.cc

@@ -21,6 +21,9 @@
 #include <windows.h>
 #include <io.h>
 #include <share.h>
+#include <locale> // For UTF-8 conversion
+#include <cassert> // For checking string size


assert.h is already included.

jhasse · 2020-08-18T20:09:07Z

src/util.cc

@@ -21,6 +21,9 @@
 #include <windows.h>
 #include <io.h>
 #include <share.h>
+#include <locale> // For UTF-8 conversion


jhasse · 2021-02-22T21:58:17Z

See #1915. Microsoft finally came to their senses and added UTF-8 support to the C runtime and the WinAPI.

This PR might still be useful to fix #1900, but maybe Microsoft will even do the right thing there and support long path names for *A functions, too.

jlonnberg · 2021-02-27T15:16:58Z

See #1915. Microsoft finally came to their senses and added UTF-8 support to the C runtime and the WinAPI.

This PR might still be useful to fix #1900, but maybe Microsoft will even do the right thing there and support long path names for *A functions, too.

Glad to hear that! I still believe that the *W functions is the way to go.
I will try to make fix the conflicts etc to make this work (been crowded at work and on parental leave so I've struggled to find the time to fix this...)

jhasse · 2021-02-28T16:34:07Z

I still believe that the *W functions is the way to go.

Why? Sounds like a huge waste of time to me, only because MS can't lift the 260 character path limit for *A functions, because ... yeah no idea why.

jlonnberg · 2021-03-10T11:14:32Z

I see your point. So basically, users on Windows 10 1903 will not need the change. Is there any need to support older versions? If not, this PR might no longer be valid...

jhasse · 2021-03-12T15:56:32Z

Is there any need to support older versions?

From my point of view, no.

jlonnberg · 2021-09-13T08:51:46Z

@jhasse Is this PR still wanted or should I just drop it?

jhasse · 2022-02-22T10:16:08Z

Closing in favor of #1915 and #2056.

Ninja UTF8 conversion: Using native windows API for conversion.

9310959

jhasse reviewed Nov 8, 2019

View reviewed changes

jhasse added the windows label Nov 8, 2019

jlonnberg added 5 commits November 11, 2019 09:45

Renamed maine to mainUTF8 and fixed some formatting.

4b1d5da

Removing trailing whitespaces.

7d287c6

Removing unneeded flag.

2acd193

Update convertCommandLine to return char**

1c0c78c

Clang formatting.

f9ff77b

jhasse reviewed Nov 12, 2019

View reviewed changes

jlonnberg added 3 commits November 13, 2019 11:49

Adding unicode compiler flags for msvc and mingw

17f0d3d

Check size of buffer before performing copying, switched from strcpy …

06b6a21

…to string::copy.

Removing unneeded variables

3fb8d1e

jhasse reviewed Nov 13, 2019

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

Update CMakeLists with jhasse's comments

e76d396

This comment was marked as abuse.

Sign in to view

tristanlabelle approved these changes Nov 13, 2019

View reviewed changes

src/util.cc Outdated Show resolved Hide resolved

src/util.cc Outdated Show resolved Hide resolved

src/util.cc Outdated Show resolved Hide resolved

src/subprocess-win32.cc Outdated Show resolved Hide resolved

src/disk_interface.cc Outdated Show resolved Hide resolved

This comment was marked as abuse.

Sign in to view