Linux/GCC Support

[Random06457]

How to setup

git clone git@github.com:Random06457/ShipWright.git -b linux
cd ShipWright
cp <path to your ROM> OTRExporter
sudo docker build . -t soh
sudo docker run --rm -it -v $(pwd):/soh soh /bin/bash

inside docker:

git clone https://github.com/ladislav-zezula/StormLib external/StormLib
cmake -B external/StormLib/build -S external/StormLib
cmake --build external/StormLib/build
cp external/StormLib/build/libstorm.a external
cp /usr/local/lib/libGLEW.a external

cd soh
make setup -j$(nproc)
make -j $(nproc)

Closes #22

Todo:

• Test on Windows
• Fix Kaleidoscope crash
• (Statically link SDL (?) )
• ( Find CXX_FILES / C_FILES automatically in libultraship/soh (?) )
• Add pulseaudio backend

[retropieuser]

Hey so trying this with a raspberry pi 4 that's running Debian Buster. I skipped the docker part as I think that's to accommodate 64bit processes and turn them into 32 bit right? As the pi is already 32 bit, but I get stuck atm. Any ideas?

edit to tidy up the log:- https://pastebin.com/3HYbUxwJ

[Random06457]

SOH currently only supports 32 bit. The point of the docker container is to be able to get the required dependencies in a less painful way (more precisely to avoid overriding existing packages with their i386 version which you probably don't want system-wise).

As for your issue, you need at least GCC 10 (though, replacing -std=c++20 with -std=c++2a in the Makefiles may work to some extend). You can also proabably remove -m32 assuming you're using a 32 bit compiler.

PS: if you have discord, it may be easier to communicate on the SOH server in the #development channel.
Also, I would also appreciate if you could send your logs in a format that doesn't take the whole github page (by uploading your logs to pastebin for instance).

[Erotemic]

I attempted to reproduce this build, but I'm getting an error in the make setup phase.

The exact procedure I followed is this:

git clone https://github.com/HarbourMasters/Shipwright.git "$HOME"/code/Shipwright
cd "$HOME"/code/Shipwright

git remote add Random06457 https://github.com/Random06457/Shipwright.git
git fetch Random06457
git checkout linux
# https://github.com/HarbourMasters/Shipwright/pull/28

# Copy ROM assets (and check they are as expected)
cp "$HOME"/data/roms/baserom_original_non_mq.z64 soh/baserom_original_non_mq.z64
echo "3c10b67a76616ae2c162def7528724cf  soh/baserom_original_non_mq.z64" > soh/baserom_original_non_mq.z64.md5
md5sum -c soh/baserom_original_non_mq.z64.md5

(cd soh && python fixbaserom.py)
(cd soh && python extract_baserom.py)

docker build . -t soh
docker run --rm -it -v "$(pwd):/soh" soh /bin/bash

Inside docker:

# Inside docker
git clone https://github.com/ladislav-zezula/StormLib external/StormLib
cmake -B external/StormLib/build -S external/StormLib
cmake --build external/StormLib/build
cp external/StormLib/build/libstorm.a external
cp /usr/local/lib/libGLEW.a external

cd soh
make setup # Fails here

Error I am getting is:

../ZAPDTR/ZAPD.out botr -se OTR
[2022-03-27 22:37:26.989] [error] [Archive.cpp:323] (2) Failed to open main mpq file /soh/OTRExporter/oot.otr.
cp ../OTRExporter/oot.otr .
make[1]: Leaving directory '/soh/soh'

Full output logs for make setup are here:
https://pastebin.com/FeVV4EcZ

Docker version is: Docker version 20.10.13, build a224086

EDIT:

I tried just running make for fun, and it did seem to produce a valid soh.elf file, which I was then able to execute on the host system! (although it does crash if I pause twice with a SegFault - I assume that is the Kaleidoscope crash?)

[Erotemic]

I ran the program with gdb and got some info about where the segfault is. I'm a bit out of my element with C++ development, so I'll paste some of the traceback and hope it helps someone who understand this a bit better:

Output is in pastebin:
https://pastebin.com/KrZJaNzS

Segfault seems to be somewhere in Resource.cpp. It happens when calling auto resShared = resMgr->GetCachedFile(hashStr); with hashStr = "textures/icon_item_static/gItemNamePanelDL".

[MegaMech]

@Erotemic Please use pastebin for long code and terminal snippets.

[Random06457]

I attempted to reproduce this build, but I'm getting an error in the make setup phase.

The exact procedure I followed is this:

git clone https://github.com/HarbourMasters/Shipwright.git "$HOME"/code/Shipwright
cd "$HOME"/code/Shipwright

git remote add Random06457 https://github.com/Random06457/Shipwright.git
git fetch Random06457
git checkout linux
# https://github.com/HarbourMasters/Shipwright/pull/28

# Copy ROM assets (and check they are as expected)
cp "$HOME"/data/roms/baserom_original_non_mq.z64 soh/baserom_original_non_mq.z64
echo "3c10b67a76616ae2c162def7528724cf  soh/baserom_original_non_mq.z64" > soh/baserom_original_non_mq.z64.md5
md5sum -c soh/baserom_original_non_mq.z64.md5

(cd soh && python fixbaserom.py)
(cd soh && python extract_baserom.py)

docker build . -t soh
docker run --rm -it -v "$(pwd):/soh" soh /bin/bash

Inside docker:

# Inside docker
git clone https://github.com/ladislav-zezula/StormLib external/StormLib
cmake -B external/StormLib/build -S external/StormLib
cmake --build external/StormLib/build
cp external/StormLib/build/libstorm.a external
cp /usr/local/lib/libGLEW.a external

cd soh
make setup # Fails here

Error I am getting is:

../ZAPDTR/ZAPD.out botr -se OTR
[2022-03-27 22:37:26.989] [error] [Archive.cpp:323] (2) Failed to open main mpq file /soh/OTRExporter/oot.otr.
cp ../OTRExporter/oot.otr .
make[1]: Leaving directory '/soh/soh'

Full output logs for make setup are here: https://pastebin.com/FeVV4EcZ

Docker version is: Docker version 20.10.13, build a224086

EDIT:

I tried just running make for fun, and it did seem to produce a valid soh.elf file, which I was then able to execute on the host system! (although it does crash if I pause twice with a SegFault - I assume that is the Kaleidoscope crash?)

Failed to open main mpq file is a mislieading log message which you can ignore (it will be removed in the future).
As for the crash you're experiencing, that's indeed what I refered to as "kaleidoscope crash".

[Erotemic]

@Random06457

Segfault seems to be somewhere in Resource.cpp. It happens when calling auto resShared = resMgr->GetCachedFile(hashStr); with hashStr = "textures/icon_item_static/gItemNamePanelDL".

I verified that if I commented out lines 48 - 55 in the destructor of libultraship/Resource.cpp the crash no longer happens. No idea if that is a sustainable fix or not. It probably introduces a leak, but the game seems to be playable

            // auto resShared = resMgr->GetCachedFile(hashStr);
            // if (resShared != nullptr)
            // {
            //    auto res = (Ship::DisplayList*)resShared.get();

            //    Gfx* gfx = (Gfx*)&res->instructions[patches[i].index];
            //    gfx->words.w1 = patches[i].origData;
            // }

edit: Played it for a little bit. Defiantly bugs.

In the Deku Tree, I'm able to cut grass as long as I'm in the same horizontal plane, regardless of the distance to the grass (this does not happen to the enemy Deku Babas). Reflecting the deku scrubs nuts happens almost instantaneously. And when I was outside the Deku tree, every swing I made hit an invisible gossip stone. Not sure if these are due to what I commented out or not.

[Ricardofriki97]

Hi , I have a problem , said it "cloning in 'ShipWright'..." "git@github.com: Permission denied (publickey)." Could not read from remote repository.

Please make sure you have the correct access permissions
and that the repository exists." can you help me please

[briaguya-ai]

Hi , I have a problem , said it "cloning in 'ShipWright'..." "git@github.com: Permission denied (publickey)." Could not read from remote repository.

Please make sure you have the correct access permissions and that the repository exists."

Looks like an issue with your github key config, you can always use https instead of git if you just want to pull the code

[Ricardofriki97]

I fix it but now I have this problem "docker build . -t soh
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=soh&target=&ulimits=null&version=1": dial unix /var/run/docker.sock: connect: permission denied" can someone help me with that problem please

[Random06457]

I fix it but now I have this problem "docker build . -t soh Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=soh&target=&ulimits=null&version=1": dial unix /var/run/docker.sock: connect: permission denied" can someone help me with that problem please

You need root perm for docker so don't forget sudo at the beginning

[Ricardofriki97]

thanks is works

[Ricardofriki97]

now I send this command "sudo docker run --rm -it -v "$(pwd):/soh" soh /bin/bash" and only have $(pwd):/soh" soh /bin/bash
root@b4a33bc640ea:/soh# and nothing more , what can I do?

[Erotemic]

@Ricardofriki97 Discord would be a better place to ask questions about basics. You'll probably get quicker support there.

[Ricardofriki97]

but in discord send me here hahahaha

[Ricardofriki97]

beside in discord is all talking about the windows version and I don´t care that version is that because I am here

[Random06457]

To makes things clear, if you're asking for support, please do that on the discord server and feel free to ping me (random#6457) if it's related to this PR. It would be better to keep this place for actual issues with the PR.

[Erotemic]

So, I've found the point where the game becomes unplayable. In the Deku Tree, when you are in the room with the big spike roller (you have to press a button to lower the water level), whenever you are on the same vertical level as the spike roller the game registers a collision with link and you get hit.

Not sure if the code I commented out has anything to do with collisions, or if it's a different bug.

[Ricardofriki97]

Is done thanks anyway and somebody help me with the problem the crash the second pause

[Ricardofriki97]

in the deku room I have not problem at all

[retropieuser]

SOH currently only supports 32 bit. The point of the docker container is to be able to get the required dependencies in a less painful way (more precisely to avoid overriding existing packages with their i386 version which you probably don't want system-wise).

As for your issue, you need at least GCC 10 (though, replacing -std=c++20 with -std=c++2a in the Makefiles may work to some extend). You can also proabably remove -m32 assuming you're using a 32 bit compiler.

PS: if you have discord, it may be easier to communicate on the SOH server in the #development channel.
Also, I would also appreciate if you could send your logs in a format that doesn't take the whole github page (by uploading your logs to pastebin for instance).

That's fair, I mean the -march is pointing to i386 as well, whereas the pi4 is -march=armv8-a+crc+simd. I imagine down the line they'll need to be an If statement for the different arm versions of the pi if SOH does get support for that.

I tried to replace -std=c++20 with -std=c++2a but still got some errors, so figured I'd try and compile GCC10 for the pi4.

Trouble is, apparently it takes about 3.5 hours, I had a go at that and got an error compilining. So I'm gonna throw in the towel for now until someone with a lot more knowledge of linux and the pi architecture. I might get the windows version up and running through box86 and wine. A very roundabout way of doing things but it's early days and I'm sure a dev with raspberry pi knowledge will be able to figure this out better than I can.

[Bloxxel64]

bash: /etc/profile.d/vte.sh: No such file or directory
bloxxel64@bloxxel64-BungieMacPro:~/Programs$ git clone git@github.com:Random06457/ShipWright.git -b linux
Cloning into 'ShipWright'...
The authenticity of host 'github.com (140.82.112.4)' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com,140.82.112.4' (ECDSA) to the list of known hosts.
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
bloxxel64@bloxxel64-BungieMacPro:~/Programs$

[Kapouett]

@Bloxxel64 try cloning with HTTP instead of SSH
git clone https://github.com/Random06457/Shipwright.git -b linux

[Quackdoc]

using docker builds fine on arch 64, doesn't work but it builds.

Soh.elf requires 32bit libs on the host system. Adding the missing libs should have it working.

im not at my PC right now. but Im pretty sure I got most of them, I was getting some kind of segfault. ill be able to get a proper log tommorow

[Random06457]

using docker builds fine on arch 64, doesn't work but it builds.

Soh.elf requires 32bit libs on the host system. Adding the missing libs should have it working.

As I said earlier, just building for 64 bit isn't enough, you'll have to actually make some code and design changes in order to get it to work on 64 bit architectures.

[Quackdoc]

using docker builds fine on arch 64, doesn't work but it builds.

Soh.elf requires 32bit libs on the host system. Adding the missing libs should have it working.

As I said earlier, just building for 64 bit isn't enough, you'll have to actually make some code and design changes in order to get it to work on 64 bit architectures.

Oh I see, I thought the 32bit build would work on 64 as long as the libs and stuff are all there. I suppose I can spin up a 32bit VM or two and try it like that.

any distros specifically that I should be testing? I would like at least on best case distro for testing then I can test a few other distros for completeness.

[Random06457]

using docker builds fine on arch 64, doesn't work but it builds.

Soh.elf requires 32bit libs on the host system. Adding the missing libs should have it working.

As I said earlier, just building for 64 bit isn't enough, you'll have to actually make some code and design changes in order to get it to work on 64 bit architectures.

Oh I see, I thought the 32bit build would work on 64 as long as the libs and stuff are all there. I suppose I can spin up a 32bit VM or two and try it like that.

any distros specifically that I should be testing? I would like at least on best case distro for testing then I can test a few other distros for completeness.

To clarify once and for all:

• Building for 32-bit and running the binary on a 32-bit architecture obviously works.
• Building for 32-bit and running the binary on a 64-bit architecture/OS that supports running 32-bit binaries (what most people are currently doing) works.
• Building for 64-bit works but running the binary doesn't.

The reason for that is that some part of the code do some unsafe computation on pointers that assume the code is being run on a 32-bit architecture.
To give an example of that, some F3DZEX commands take an address to reference some data like textures, matrices, vertices, etc. Since these addresses are encoded on 32 bits in the command list format, the code ends up doing pointer truncation on 64-bit architectures, which will likely result in a segfault when dereferencing these pointers. So in order to fix that you'll have to make some breaking changes to:

• The command list exporter in ZAPDTR
• Every piece of code in the game that encodes the problematic commands
• The command list parser logic in Fast3D

And this is only one example, there are probably other systems in the code that do pointer truncation for whatever reason which cause the same issue.

[Quackdoc]

Maybe I was misleading, apologies if so. I am making a 32bit build using the docker, trying to run it on a 64bit arch install that has the necessary (to my knowledge) 32bit libraries below is the backtrace of the crash, and attached is the terminal typescript.

#0  0x00000000 in ?? ()
#1  0xf7d36d71 in ?? () from /usr/lib32/libSDL2-2.0.so.0
#2  0xf7d02f28 in ?? () from /usr/lib32/libSDL2-2.0.so.0
#3  0xf7d6c376 in ?? () from /usr/lib32/libSDL2-2.0.so.0
#4  0xf7c7a0cb in ?? () from /usr/lib32/libSDL2-2.0.so.0
#5  0xf7d27389 in ?? () from /usr/lib32/libSDL2-2.0.so.0
#6  0xf7d2cadc in ?? () from /usr/lib32/libSDL2-2.0.so.0
#7  0xf7d2e28b in ?? () from /usr/lib32/libSDL2-2.0.so.0
#8  0xf7c795d0 in ?? () from /usr/lib32/libSDL2-2.0.so.0
#9  0xf7c7996a in ?? () from /usr/lib32/libSDL2-2.0.so.0
#10 0xf7c79c52 in ?? () from /usr/lib32/libSDL2-2.0.so.0
#11 0x56dd7b24 in gfx_sdl_handle_events () at libultraship/Lib/Fast3D/gfx_sdl2.cpp:232
#12 0x56d120da in gfx_start_frame () at libultraship/Lib/Fast3D/gfx_pc.cpp:2696
#13 0x56dbee88 in Ship::Window::StartFrame (this=0x57294f5c) at libultraship/Window.cpp:283
#14 0x56c9d095 in Graph_StartFrame () at soh/OTRGlobals.cpp:117
#15 0x56839d89 in RunFrame () at src/code/graph.c:477
#16 0x56dd7a14 in gfx_sdl_main_loop (run_one_game_iter=0x56839c3c <RunFrame>) at libultraship/Lib/Fast3D/gfx_sdl2.cpp:199
#17 0x56dbf03d in Ship::Window::MainLoop (this=0x57294f5c, MainFunction=0x56839c3c <RunFrame>) at libultraship/Window.cpp:323
#18 0x56c9d005 in Graph_ProcessFrame (run_one_game_iter=0x56839c3c <RunFrame>) at soh/OTRGlobals.cpp:113
#19 0x56839eb8 in Graph_ThreadEntry (arg0=0x0) at src/code/graph.c:508
#20 0x5683bd85 in Main (arg=0x0) at src/code/main.c:117
#21 0x5683b9b5 in main (argc=1, argv=0xffffcbb4) at src/code/main.c:46

typescript.log

I am using the file with this hash cee6bc3c2a634b41728f2af8da54d9bf8cc14099

[vaguerant]

At GaryOderNichts#7 (the Wii U port), I filed an issue about some push blocks failing to play the "solved" jingle and save their final positions after you've pushed them. Gary's fork is based on the work in this PR. Thanks to @Baoulettes, we've confirmed this seems to be a GCC issue rather than Wii U-specific. The bug does not occur on Windows/MSVC builds.

In the issue at Gary's fork, I identified two push blocks with the issue:

• 79: Water Temple 1F, main room, push block at left
• 65: Gerudo Training Ground, push block in the back left room

I also just discovered a third:

• 81: Spirit Temple, adult, entrance room, push block at right

The Water Temple and Spirit Temple blocks are the easiest ones to reach as they appear in the first rooms of their respective dungeons. In all three cases, the blocks are intended to be pushed into pits, play the solved jingle, and afterwards stay in that position indefinitely. Instead, they don't play the jingle and reset to their starting positions as soon as the player leaves the room.

This issue isn't game-breaking in any case, but the Water Temple and Gerudo Training Grounds push blocks do trap the player behind them--in normal gameplay, the only exit in their respective rooms is to walk back over the "solved" push block, but instead their pits are empty and the blocks are obstructing the exits. The Spirit Temple has alternative exit routes which don't require the player to return via the same route through which they entered. For the Water Temple and Gerudo Training Grounds blocks, the only workaround is to save and reset to return to the dungeon entrance. Or use Farore's Wind, I guess.

EDIT: Gary and Baoulettes both re-tested on Linux/GCC and could not repro again, so it seems likely this was indeed a Wii U bug specifically. Gary has fixed it on their end with GaryOderNichts/Shipwright@d08fff8. It was a floating point rounding issue.

[Kenix3]

Merge pending conflict resolution. Great job on this random!!!