Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Windows build #42

Closed
wants to merge 22 commits into from
Closed

Conversation

bilderbuchi
Copy link

@bilderbuchi bilderbuchi commented Jun 14, 2021

Checklist

  • Used a personal fork of the feedstock to propose changes
  • Bumped the build number (if the version is unchanged)
  • Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
  • Ensured the license file is being packaged.
  • Try to use - {{ compiler('m2w64_cxx') }} # [win] again and re-render the recipe afterwards!

Let's see how far we get in adding a Windows build of gdb.
See #6.

@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@bilderbuchi bilderbuchi changed the title Windows build [WIP] Windows build Jun 14, 2021
@bilderbuchi bilderbuchi marked this pull request as draft June 14, 2021 18:26
@bilderbuchi
Copy link
Author

@conda-forge-admin , please rerender

@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

  • Selectors are suggested to take a <two spaces>#<one space>[<expression>] form. See lines [27]

@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@bilderbuchi
Copy link
Author

bilderbuchi commented Jun 15, 2021

um, I'm getting up to the make stage now, but I get "configure: error: C compiler cannot create executables" (link)

No idea what to do about that, does anybody have input here?

@bilderbuchi bilderbuchi mentioned this pull request Jun 25, 2021
4 tasks
Start bld.bat using snippets from AnacondaRecipes [1]
The script calls build.sh to reduce duplication.
Remove packages unavailable on Windows.
Add necessary m2 packages.

[1] https://github.com/AnacondaRecipes/conda-feedstock/blob/master/recipe/bld.bat
@bilderbuchi
Copy link
Author

@conda-forge-admin, please rerender

conda-forge-linter and others added 2 commits July 1, 2021 19:16
@bilderbuchi
Copy link
Author

bilderbuchi commented Jul 1, 2021

Hm, this fails with

conda.exceptions.ResolvePackageNotFound: 
  - ncurses

although I specifically first excluded ncurses, then replaced it with m2-ncurses (both approaches fail).

I can't recognize where/why it needs ncurses. The log output before even seems to suggest that conda's finished, but then another solve fails?!

Curiously enough, this did not occur in the previous incarnation of this PR. Maybe a change in the recipe rerender template?

@bilderbuchi
Copy link
Author

@klauer: interestingly, the re-render above (181521f) apparently broke (only) the macos py39 build, although no mac files seems to have been touched?

@klauer
Copy link
Member

klauer commented Jul 1, 2021

Strange - diffing the working build output to the non-working one, I don't see many differences:
This PR's build:

  • conda-forge-ci-setup-3.9.4
  • Test phase fails with the gdb output During startup program terminated with signal ?, Unknown signal. which happens before even a thread gets created (*)
  • Doesn't touch any of the travis config as you noted
(*) Test diff

Working:

Developer mode is already enabled.
YES (0)
[New Thread 0x1503 of process 40618]
[New Thread 0x2803 of process 40618]
warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/Objects-normal/x86_64/darwin_vers.o': can't open to read symbols: No such file or directory.
warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/Objects-normal/x86_64/dirstat.o': can't open to read symbols: No such file or directory.
warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/Objects-normal/x86_64/dirstat_collection.o': can't open to read symbols: No such file or directory.
warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/Objects-normal/x86_64/init.o': can't open to read symbols: No such file or directory.
warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/Objects-normal/x86_64/variant.o': can't open to read symbols: No such file or directory.
Hello, World!
[Inferior 1 (process 40618) exited normally]

Not working:

Developer mode is already enabled.
YES (0)
During startup program terminated with signal ?, Unknown signal.
[New Thread 0x1503 of process 40602]
Tests failed for gdb-10.2-py39h593ab31_0.tar.bz2 - moving package to /Users/travis/miniforge3/conda-bld/broken
WARNING:conda_build.build:Tests failed for gdb-10.2-py39h593ab31_0.tar.bz2 - moving package to /Users/travis/miniforge3/conda-bld/broken
WARNING conda_build.build:tests_failed(2955): Tests failed for gdb-10.2-py39h593ab31_0.tar.bz2 - moving package to /Users/travis/miniforge3/conda-bld/broken

Working build

  • conda-forge-ci-setup-3.9.5 (no significant differences as far as I can tell, recipe here)
  • Everything else looks the same

I'd imagine the tests aren't so flaky as the 10.2 PR merged and built without issue...

@bilderbuchi
Copy link
Author

Curiously, on the next run, both macos-py36 and macos-py39 failed, although again no macos config was touched. Maybe something is flaky after all.

Btw, easter-egg time, gdb still contains python 2 code, although that does not fail the build 😁

@bilderbuchi
Copy link
Author

bilderbuchi commented Jul 2, 2021

Meanwhile, there's conda-forge-ci-setup 3.9.6 released, so
@conda-forge-admin, please rerender

@github-actions
Copy link
Contributor

github-actions bot commented Jul 2, 2021

Hi! This is the friendly automated conda-forge-webservice.
I tried to rerender for you, but it looks like there was nothing to do.

2 similar comments
@github-actions
Copy link
Contributor

github-actions bot commented Jul 2, 2021

Hi! This is the friendly automated conda-forge-webservice.
I tried to rerender for you, but it looks like there was nothing to do.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 2, 2021

Hi! This is the friendly automated conda-forge-webservice.
I tried to rerender for you, but it looks like there was nothing to do.

@phil-blain
Copy link
Contributor

I'll try to help if I can for the Windows build, but I have close to zero experience on Windows.

Regarding the macOS Travis run that seems flaky, upon re-reading the GDB bugzilla entry linked in recipe/README, I think that I would have to update the patch that we use for macOS. I'll try to do that some time. In the meantime I think we can ignore macOS failures in this here PR, at least the During startup program terminated with signal ?, Unknown signal. error.

Regarding the latest macOS py36 failure, I see this at the end:

Solving environment: ...working... failed
failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform osx-64: {'gdb==10.2=py36h7a1bca3_1'}
WARNING:conda_build.build:failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform osx-64: {'gdb==10.2=py36h7a1bca3_1'}
WARNING conda_build.build:test(2859): failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform osx-64: {'gdb==10.2=py36h7a1bca3_1'}

I'm not sure about this one. I don't see anything wrong before that. There's a lot of "is not an ELF" lines but that might just be conda-build acting out, I'm not sure. It's the same thing for the macOS py39 failure in 181521f...

@bilderbuchi
Copy link
Author

bilderbuchi commented Aug 3, 2021

yeah, I think we both hit a temporary CI breakdown, which has since resolved. I managed to get over some more roadblocks, and checked out the changes of that PR, but at this point I think my problems have a different source (\ in windows paths tripping up this dwarf function), but it's hard to troubleshoot (and find the time+energy for that).

@bilderbuchi
Copy link
Author

Specifically, I suspect I need to add one more env vars to the cygpath conversion done here, but I currently don't know how to find out which path is the culprit. I think the warnings about escapes surrounding the error linked above could each stem from the initial letters of directories in the affected path, but I have not yet tried to decipher that.

@bilderbuchi
Copy link
Author

bilderbuchi commented Aug 20, 2021

The DEBUGDIR path in the lines 1 and 2 (and what follows in the same line is suspiciously similar to the pattern of invalid escape sequence warnings.
I suspect that this somehow (how?!) mangles forward-slash delimited paths as in the command line invocation into backslash delimited paths (with some expansion as a capital \L is also in there?), which then trigger all these invalid escape sequence warnings?

It's probably that --with-separate-debug-dir flag, I'll have to find out how to set that correctly on Windows.

@bilderbuchi
Copy link
Author

bilderbuchi commented Aug 21, 2021

OK, I disabled that flag on Windows, now it fails because it does not find netdb.h, which is apparently not included in mingw-64:

D:/bld/gdb_1629543547423/work/gdbsupport/netstuff.cc:28:19: fatal error: netdb.h: No such file or directory

Curiously enough, above that point it checks for the presence of netdb.h and correctly recognizes that it's not available, so I don't know why it stumbles over that later.

I looked at the various patches linked above but noticed nothing that seemed applicable.

@bilderbuchi
Copy link
Author

Since gdb is not available from conda-forge for Windows, and it seems to me it should be, I have been trying to get that to build.
Apparently, I don't know what I'm doing, because I've been banging my head against this over the summer. At this point I'm (again) stuck.

Maybe some Windows/m2w64/mingw/gcc experts can take a look and help out getting gdb built? @conda-forge/help-c-cpp @conda-forge/core

@bilderbuchi
Copy link
Author

bilderbuchi commented Aug 24, 2021

My reasoning for going the route of a slim bld.bat shim that calls build.sh was to minimize the additional maintenance effort for the Windows build by not introducing another build script. I saw that approach in another package, so it seemed reasonable, but maybe it isn't?

@jaimergp
Copy link
Member

jaimergp commented Aug 25, 2021

Looks like a missing header: netdb.h. This seems to be not part of MingW, so you'll need to patch it to replace it by a Windows equivalent. Source 1, source 2, source 3.

Edit: oops sorry didn't read the full conversation. You might have tried this already 😬 . Anyway, you can see a similar patch for the MingW Python here. Worth a try? I don't know why they are not using it on their GDB build though.

Edit 2: It looks like configure should handle this but it does not. The offending file will check for the variable USE_WIN32API to use the Windows API or the Unix one, so I guess that one is not being set appropriately. Can you force define it in the calling script?

recipe/meta.yaml Outdated
- make >=3.82
- {{ compiler('c') }} # [not win]
- {{ compiler('cxx') }} # [not win]
- m2w64-gcc # [win]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a compiler function for m2w64 too. See here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to rerender the recipe for this to have an effect.

Copy link
Author

@bilderbuchi bilderbuchi Aug 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I already tried this before: - {{ compiler('m2w64_cxx') }} # [win] but it didn't work (c639a13). Argh, no use checking the log, when does CI start throwing away logs??
Come to think of it, I don't think I re-rendered as I did not know that that's a prerequisite, I'll try this again...

@bilderbuchi
Copy link
Author

It looks like configure should handle this but it does not.

Yeah, right? That's what I thought, too. That USE_WIN32API thing looks promising!

It's funny, mingw-64 actually has gdb available (in fact, that's what I'm using in the meantime, but not installed via the conda ecosystem), I just can't find out how the manage to build it... :-/

@bilderbuchi
Copy link
Author

hm, this could be the build script...

Otherwise, the gdb configure script does not correctly
choose ws2tcpip.h instead of netdb.h et al.
@jaimergp
Copy link
Member

jaimergp commented Aug 25, 2021

I think USE_WIN32API needs to be a preprocessor variable (CFLAGS containing -DUSE_WIN32API), not an env var, but it's been a while since I have done Windows stuff.

edit -- hah, ok, you pushed that commit as I posted this :D

@bilderbuchi
Copy link
Author

I'm currently on it :D

@bilderbuchi
Copy link
Author

bilderbuchi commented Aug 25, 2021

After waiting for the sacred Interval, you turn another page in the mysterious book. Like the tomes around you in the great library, it is full of inscrutable and arcane symbols. As the sparkling dust settles, you glimpse yet another foe. The horror that arises from the page is...

checking whether strtoul is declared...   CXXLD  gdbreplay.exe
yes
gdbreplay.o:gdbreplay.cc:(.text+0xf7): undefined reference to `__imp_closesocket'
gdbreplay.o:gdbreplay.cc:(.text+0x224): undefined reference to `__imp_WSAStartup'
gdbreplay.o:gdbreplay.cc:(.text+0x276): undefined reference to `__imp_getaddrinfo'
gdbreplay.o:gdbreplay.cc:(.text+0x294): undefined reference to `gai_strerrorA'
gdbreplay.o:gdbreplay.cc:(.text+0x35d): undefined reference to `__imp_socket'
gdbreplay.o:gdbreplay.cc:(.text+0x3d3): undefined reference to `__imp_setsockopt'
gdbreplay.o:gdbreplay.cc:(.text+0x482): undefined reference to `__imp_bind'
gdbreplay.o:gdbreplay.cc:(.text+0x4db): undefined reference to `__imp_listen'
gdbreplay.o:gdbreplay.cc:(.text+0x511): undefined reference to `__imp_accept'
gdbreplay.o:gdbreplay.cc:(.text+0x56a): undefined reference to `__imp_setsockopt'
gdbreplay.o:gdbreplay.cc:(.text+0x5a6): undefined reference to `__imp_setsockopt'
gdbreplay.o:gdbreplay.cc:(.text+0x5e2): undefined reference to `__imp_getnameinfo'
gdbreplay.o:gdbreplay.cc:(.text+0x641): undefined reference to `__imp_closesocket'
gdbreplay.o:gdbreplay.cc:(.rdata$.refptr.in6addr_any[.refptr.in6addr_any]+0x0): undefined reference to `in6addr_any'
../gdbsupport/libgdbsupport.a(netstuff.o):netstuff.cc:(.text+0x57): undefined reference to `__imp_freeaddrinfo'
collect2.exe: error: ld returned 1 exit status

⚔️ 🧙

Roll a D20 for monster lore... 12... it seems to be a Linking Demon!

@bilderbuchi
Copy link
Author

@conda-forge-admin, please rerender

@bilderbuchi
Copy link
Author

OK, LDFLAGS seems to end up withe reasonable values 'LDFLAGS= -lwsock32 -lws2_32 -Wl,--stack,12582912', but still ld fails to find these winsock(?) symbols. Dunno what's going on right now.

@phil-blain
Copy link
Contributor

phil-blain commented Aug 25, 2021

You could try doing make V=1, this will output the full commands to the log file instead of CXX <file>.o, CXXLD <exe>, etc ... Also adding -O (capital O) to the make invocation might help with the intermingled lines (in theory it should, not sure it will work with the CI layer log redirection).

@bilderbuchi
Copy link
Author

bilderbuchi commented Aug 29, 2021

Thanks for those tips! This is the relevant log section

2021-08-29T20:44:46.4356997Z g++  -std=gnu++17    -I. -I/d/bld/gdb_1630268986132/work/gdbserver -I/d/bld/gdb_1630268986132/work/gdbserver/../gdb/regformats -I/d/bld/gdb_1630268986132/work/gdbserver/.. -I/d/bld/gdb_1630268986132/work/gdbserver/../include -I/d/bld/gdb_1630268986132/work/gdbserver/../gdb -I/d/bld/gdb_1630268986132/work/gdbserver/../gnulib/import -I../gnulib/import -I/d/bld/gdb_1630268986132/work/gdbserver/.. -I.. -I./../intl -I/d/bld/gdb_1630268986132/_h_env/include -DUSE_WIN32API -pthread -Wall -Wpointer-arith -Wno-unused -Wunused-value -Wunused-variable -Wunused-function -Wno-switch -Wno-char-subscripts -Wempty-body -Wunused-but-set-parameter -Wunused-but-set-variable -Wno-sign-compare -Wno-error=maybe-uninitialized -Wsuggest-override -Wmissing-declarations -Wstrict-null-sentinel -Wformat -Wformat-nonliteral  -DGDBSERVER -lwsock32 -lws2_32 -Wl,--stack,12582912  \
2021-08-29T20:44:46.4358761Z 	-o gdbreplay.exe gdbreplay.o utils.o version.o  \
2021-08-29T20:44:46.4359436Z 	../gdbsupport/libgdbsupport.a ../gnulib/import/libgnu.a ../libiberty/libiberty.a ./../intl/libintl.a -liconv \
2021-08-29T20:44:46.4360053Z 	
2021-08-29T20:44:46.4360641Z gdbreplay.o:gdbreplay.cc:(.text+0xf7): undefined reference to `__imp_closesocket'
2021-08-29T20:44:46.4361495Z gdbreplay.o:gdbreplay.cc:(.text+0x224): undefined reference to `__imp_WSAStartup'
2021-08-29T20:44:46.4362202Z gdbreplay.o:gdbreplay.cc:(.text+0x276): undefined reference to `__imp_getaddrinfo'
2021-08-29T20:44:46.4363098Z gdbreplay.o:gdbreplay.cc:(.text+0x294): undefined reference to `gai_strerrorA'
2021-08-29T20:44:46.4363768Z gdbreplay.o:gdbreplay.cc:(.text+0x35d): undefined reference to `__imp_socket'
2021-08-29T20:44:46.4364492Z gdbreplay.o:gdbreplay.cc:(.text+0x3d3): undefined reference to `__imp_setsockopt'
2021-08-29T20:44:46.4365190Z gdbreplay.o:gdbreplay.cc:(.text+0x482): undefined reference to `__imp_bind'
2021-08-29T20:44:46.4365855Z gdbreplay.o:gdbreplay.cc:(.text+0x4db): undefined reference to `__imp_listen'
2021-08-29T20:44:46.4366680Z gdbreplay.o:gdbreplay.cc:(.text+0x511): undefined reference to `__imp_accept'
2021-08-29T20:44:46.4367391Z gdbreplay.o:gdbreplay.cc:(.text+0x56a): undefined reference to `__imp_setsockopt'
2021-08-29T20:44:46.4368068Z gdbreplay.o:gdbreplay.cc:(.text+0x5a6): undefined reference to `__imp_setsockopt'
2021-08-29T20:44:46.4368789Z gdbreplay.o:gdbreplay.cc:(.text+0x5e2): undefined reference to `__imp_getnameinfo'
2021-08-29T20:44:46.4369609Z gdbreplay.o:gdbreplay.cc:(.text+0x641): undefined reference to `__imp_closesocket'
2021-08-29T20:44:46.4370351Z gdbreplay.o:gdbreplay.cc:(.rdata$.refptr.in6addr_any[.refptr.in6addr_any]+0x0): undefined reference to `in6addr_any'
2021-08-29T20:44:46.4371166Z ../gdbsupport/libgdbsupport.a(netstuff.o):netstuff.cc:(.text+0x57): undefined reference to `__imp_freeaddrinfo'
2021-08-29T20:44:46.4371878Z collect2.exe: error: ld returned 1 exit status
2021-08-29T20:44:46.4372503Z Makefile:362: recipe for target 'gdbreplay.exe' failed
2021-08-29T20:44:46.4373101Z make[2]: *** [gdbreplay.exe] Error 1

gdbreplay.o gets compiled here (apparently without mishap)

2021-08-29T20:44:28.5697995Z g++  -std=gnu++17    -I. -I/d/bld/gdb_1630268986132/work/gdbserver -I/d/bld/gdb_1630268986132/work/gdbserver/../gdb/regformats -I/d/bld/gdb_1630268986132/work/gdbserver/.. -I/d/bld/gdb_1630268986132/work/gdbserver/../include -I/d/bld/gdb_1630268986132/work/gdbserver/../gdb -I/d/bld/gdb_1630268986132/work/gdbserver/../gnulib/import -I../gnulib/import -I/d/bld/gdb_1630268986132/work/gdbserver/.. -I.. -I./../intl -I/d/bld/gdb_1630268986132/_h_env/include -DUSE_WIN32API -pthread -Wall -Wpointer-arith -Wno-unused -Wunused-value -Wunused-variable -Wunused-function -Wno-switch -Wno-char-subscripts -Wempty-body -Wunused-but-set-parameter -Wunused-but-set-variable -Wno-sign-compare -Wno-error=maybe-uninitialized -Wsuggest-override -Wmissing-declarations -Wstrict-null-sentinel -Wformat -Wformat-nonliteral  -DGDBSERVER -c -o gdbreplay.o -MT gdbreplay.o -MMD -MP -MF ./.deps/gdbreplay.Tpo /d/bld/gdb_1630268986132/work/gdbserver/gdbreplay.cc

The missing references (WSAStartup etc) shown above are behind #ifdef USE_WIN32API guards in gdbreplay.c, so the -DUSE_WIN32API seems to work in order. It just cannot seem to find the library linked with -lwsock32 -lws2_32, I don't know why.

@phil-blain
Copy link
Contributor

Copying the linking command with some newlines and commands added:

g++  -std=gnu++17    \
# Includes
-I. -I/d/bld/gdb_1630268986132/work/gdbserver \
-I/d/bld/gdb_1630268986132/work/gdbserver/../gdb/regformats \
-I/d/bld/gdb_1630268986132/work/gdbserver/.. \
-I/d/bld/gdb_1630268986132/work/gdbserver/../include \
-I/d/bld/gdb_1630268986132/work/gdbserver/../gdb \
-I/d/bld/gdb_1630268986132/work/gdbserver/../gnulib/import \
-I../gnulib/import -I/d/bld/gdb_1630268986132/work/gdbserver/.. \
-I.. -I./../intl -I/d/bld/gdb_1630268986132/_h_env/include \
# Preprocessor
-DUSE_WIN32API -pthread \
# Warnings
-Wall -Wpointer-arith -Wno-unused -Wunused-value -Wunused-variable -Wunused-function -Wno-switch -Wno-char-subscripts -Wempty-body -Wunused-but-set-parameter -Wunused-but-set-variable -Wno-sign-compare -Wno-error=maybe-uninitialized -Wsuggest-override -Wmissing-declarations -Wstrict-null-sentinel -Wformat -Wformat-nonliteral  \
# More preprocessor, your added LDFLAGS
-DGDBSERVER -lwsock32 -lws2_32 -Wl,--stack,12582912  \
# Object files
-o gdbreplay.exe gdbreplay.o utils.o version.o  \
# Static libs
../gdbsupport/libgdbsupport.a ../gnulib/import/libgnu.a ../libiberty/libiberty.a ./../intl/libintl.a \
# Shared libs
-liconv 	

According to this SO answer, the -lwsock32 -lws2_32 have to come after the object files. I think that's what you'd want to achieve, somehow.

Looking briefly at the code, I think this should all work correctly, without having to set USE_WIN32API manually.
Here is an excerpt from gdb/configure:

  WIN32APILIBS=
  case ${host} in
    *mingw32*)

$as_echo "#define USE_WIN32API 1" >>confdefs.h

      WIN32APILIBS="-lws2_32"
      ;;
  esac

So I think the problem is that host is not what this configure bit expects, in fact configure says it's: x86_64-pc-mingw64 (which I think is correct). So maybe a simple fix would be to patch this configure (and maybe others) to use *mingw* instead of *mingw32* ?

@bilderbuchi
Copy link
Author

bilderbuchi commented Sep 25, 2023

I'm closing this, this is hopelessly stale by now. Someone else, feel free to reopen/pick up where I left.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants