ARROW-4599: [C++] Add support for system GFlags #3675

kou · 2019-02-17T01:45:58Z

This is needed to build tests with MinGW.

kou · 2019-02-17T01:52:31Z

ci/travis_before_script_cpp.sh

@@ -53,7 +53,8 @@ CMAKE_COMMON_FLAGS="\
 -DARROW_NO_DEPRECATED_API=ON \
 -DARROW_EXTRA_ERROR_CONTEXT=ON"
 CMAKE_LINUX_FLAGS=""
-CMAKE_OSX_FLAGS=""
+CMAKE_OSX_FLAGS="\
+-DARROW_GFLAGS_USE_SHARED=OFF"


Because @rpath can't be resolved with GFlags installed by conda-forge.

https://travis-ci.org/kou/arrow/jobs/494364820#L1993

2/111 Test #1: plasma-serialization_tests ...................***Failed 1.36 sec Running plasma-serialization_tests, redirecting output into /Users/travis/build/kou/arrow/cpp-build/build/test-logs/plasma-serialization_tests.txt (attempt 1/1) dyld: Library not loaded: @rpath/libgflags.2.2.dylib Referenced from: /Users/travis/build/kou/arrow/cpp-install/lib/libarrow.13.dylib Reason: image not found

We will be able to resolve it by running each test with DYLD_LIBRARY_PATH environment variable:

DYLD_LIBRARY_PATH=${MINICONDA}/lib ${ARROW_BUILD_TYPE}/plasma-serialization_tests

Because DYLD_LIBRARY_PATH can affect only the child process.

It's not useful. So I use static library than shared library.

This is working for all other libraries? Then there must be something wrong with libgflags.

Other libraries are used as static library. So they work.

gflags-config.cmake provides @rpath/libgflags.2.2.dylib for IMPORTED_LOCATION. If gflags-config.cmake doesn't use @rpath, this will not be caused.

Generally, using @rpath isn't bad. GFlags installed from conda-forge can't assume where it's installed. So I think that using @rpath is reasonable for this case.

We will be able to resolve @rpath in our FindGFlags.cmake. This is another solution.

We can specify DYLD_LIBRARY_PATH to resolve @rpath in cpp/build-support/run-test.sh. This is also another solution.

kou · 2019-02-17T21:31:41Z

A job on AppVeyor is failed:

https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/22428811/job/p36y88oj2ja8on1t#L1236

gflags_static.lib(gflags.cc.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0' in json-integration-test.cc.obj
gflags_static.lib(gflags.cc.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MTd_StaticDebug' doesn't match value 'MT_StaticRelease' in json-integration-test.cc.obj
gflags_static.lib(gflags_reporting.cc.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0' in json-integration-test.cc.obj
gflags_static.lib(gflags_reporting.cc.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MTd_StaticDebug' doesn't match value 'MT_StaticRelease' in json-integration-test.cc.obj
gflags_static.lib(gflags_completions.cc.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0' in json-integration-test.cc.obj
gflags_static.lib(gflags_completions.cc.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MTd_StaticDebug' doesn't match value 'MT_StaticRelease' in json-integration-test.cc.obj
libcpmtd.lib(stdthrow.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0' in json-integration-test.cc.obj
libcpmtd.lib(stdthrow.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MTd_StaticDebug' doesn't match value 'MT_StaticRelease' in json-integration-test.cc.obj
   Creating library release\arrow-json-integration-test.lib and object release\arrow-json-integration-test.exp
LINK : warning LNK4098: defaultlib 'LIBCMTD' conflicts with use of other libs; use /NODEFAULTLIB:library
gflags_static.lib(gflags.cc.obj) : error LNK2019: unresolved external symbol _calloc_dbg referenced in function "char * __cdecl std::_Maklocstr<char>(char const *,char *,struct _Cvtvec const &)" (??$_Maklocstr@D@std@@YAPEADPEBDPEADAEBU_Cvtvec@@@Z)
gflags_static.lib(gflags_completions.cc.obj) : error LNK2001: unresolved external symbol _calloc_dbg
gflags_static.lib(gflags.cc.obj) : error LNK2019: unresolved external symbol _free_dbg referenced in function "private: void __cdecl std::_Yarn<char>::_Tidy(void)" (?_Tidy@?$_Yarn@D@std@@AEAAXXZ)
gflags_static.lib(gflags_completions.cc.obj) : error LNK2001: unresolved external symbol _free_dbg
gflags_static.lib(gflags.cc.obj) : error LNK2019: unresolved external symbol _malloc_dbg referenced in function "public: static void * __cdecl std::_Crt_new_delete::operator new(unsigned __int64,struct std::nothrow_t const &)" (??2_Crt_new_delete@std@@SAPEAX_KAEBUnothrow_t@1@@Z)
gflags_static.lib(gflags_completions.cc.obj) : error LNK2001: unresolved external symbol _malloc_dbg
gflags_static.lib(gflags.cc.obj) : error LNK2019: unresolved external symbol _CrtDbgReportW referenced in function "public: bool __cdecl std::_Tree_const_iterator<class std::_Tree_val<struct std::_Tree_simple_types<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const ,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > > > > >::operator==(class std::_Tree_const_iterator<class std::_Tree_val<struct std::_Tree_simple_types<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const ,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > > > > > const &)const " (??8?$_Tree_const_iterator@V?$_Tree_val@U?$_Tree_simple_types@U?$pair@$$CBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@V12@@std@@@std@@@std@@@std@@QEBA_NAEBV01@@Z)
gflags_static.lib(gflags_reporting.cc.obj) : error LNK2001: unresolved external symbol _CrtDbgReportW
gflags_static.lib(gflags_completions.cc.obj) : error LNK2001: unresolved external symbol _CrtDbgReportW
libcpmtd.lib(stdthrow.obj) : error LNK2001: unresolved external symbol _CrtDbgReportW
release\arrow-json-integration-test.exe : fatal error LNK1120: 4 unresolved externals

Using vendored version for this case is expected. Build type may be conflicted. I need to look into this.

kou · 2019-02-18T07:18:05Z

ci/appveyor-cpp-build.bat

@@ -44,6 +44,7 @@ if "%JOB%" == "Static_Crt_Build" (
  cmake --build . --config Debug || exit /B
  ctest --output-on-failure -j2 || exit /B
  popd
+  rmdir /S /Q cpp\build-debug


Because find_package(gflags CONFIG) can find GFlags in cpp\build-debug\gflags_ep-prefix\.

kou · 2019-02-18T07:25:53Z

CI is passed.
Now, this is ready to review.

If old build directory exists, CMake finds <package>-config.cmake from old build directory. It's incompatible with new configuration.

Change-Id: Ied430fc43304f26e7d8490fbb3818ad4e9ceb510

wesm

+1. I rebased, will merge once the build passes

kou · 2019-02-19T06:06:00Z

CI is green. I'll merge this.

This is needed to build tests with MinGW. Author: Kouhei Sutou <kou@clear-code.com> Author: Wes McKinney <wesm+git@apache.org> Closes apache#3675 from kou/cpp-system-gflags and squashes the following commits: 38f910f <Wes McKinney> Fix rebase artifact 2caf588 <Kouhei Sutou> Define GFLAGS_IS_A_DLL by target 0f43fc9 <Kouhei Sutou> Don't use vendored GFlags when GFLAGS_HOME is specified ca0f67c <Kouhei Sutou> Remove build directory entirely before new configuration build 9516d2f <Kouhei Sutou> GFLAGS_STATIC isn't defined by CONFIG mode 34cb1a6 <Kouhei Sutou> Add support for system GFlags

kou commented Feb 17, 2019

View reviewed changes

kou mentioned this pull request Feb 17, 2019

Please allow to use external packages for all dependencies (ex. zstd, flatbuffers, etc) to facilitate packaging #3668

Closed

kou force-pushed the cpp-system-gflags branch 3 times, most recently from 884a418 to 7ce098c Compare February 18, 2019 06:18

kou commented Feb 18, 2019

View reviewed changes

wesm force-pushed the master branch from 3088183 to 0c6b2d2 Compare February 18, 2019 19:34

kou and others added 6 commits February 18, 2019 19:02

[C++] Add support for system GFlags

34cb1a6

GFLAGS_STATIC isn't defined by CONFIG mode

9516d2f

Remove build directory entirely before new configuration build

ca0f67c

If old build directory exists, CMake finds <package>-config.cmake from old build directory. It's incompatible with new configuration.

Don't use vendored GFlags when GFLAGS_HOME is specified

0f43fc9

Define GFLAGS_IS_A_DLL by target

2caf588

Fix rebase artifact

38f910f

Change-Id: Ied430fc43304f26e7d8490fbb3818ad4e9ceb510

wesm force-pushed the cpp-system-gflags branch from 7ce098c to 38f910f Compare February 19, 2019 01:03

wesm approved these changes Feb 19, 2019

View reviewed changes

kou mentioned this pull request Feb 19, 2019

ARROW-4297: [C++] Fix build error with MinGW-w64 32-bit #3693

Closed

kou closed this in 24a98e4 Feb 19, 2019

kou deleted the cpp-system-gflags branch February 20, 2019 00:26

asfimport mentioned this pull request Feb 22, 2019

[C++] Add support for system GFlags #21140

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-4599: [C++] Add support for system GFlags #3675

ARROW-4599: [C++] Add support for system GFlags #3675

kou commented Feb 17, 2019 •

edited

Loading

kou Feb 17, 2019

xhochy Feb 18, 2019

kou Feb 18, 2019

kou commented Feb 17, 2019

kou Feb 18, 2019

kou commented Feb 18, 2019

wesm left a comment

kou commented Feb 19, 2019

ARROW-4599: [C++] Add support for system GFlags #3675

ARROW-4599: [C++] Add support for system GFlags #3675

Conversation

kou commented Feb 17, 2019 • edited Loading

kou Feb 17, 2019

Choose a reason for hiding this comment

xhochy Feb 18, 2019

Choose a reason for hiding this comment

kou Feb 18, 2019

Choose a reason for hiding this comment

kou commented Feb 17, 2019

kou Feb 18, 2019

Choose a reason for hiding this comment

kou commented Feb 18, 2019

wesm left a comment

Choose a reason for hiding this comment

kou commented Feb 19, 2019

kou commented Feb 17, 2019 •

edited

Loading