Implement a stub pthreads library for `THREAD_MODEL=single` #518

ArcaneNibble · 2024-07-18T02:54:55Z

This patch series first starts with a number of commits stubbing out functions in the existing THREAD_model=posix code. According to "The Open Group Base Specifications Issue 7, 2018 edition", there are a number of mandatory functions which have not been provided. There are also some optional functions that have been partially provided in a not-useful way (e.g. get but no set function). For these, I have chosen to clean them up and remove the get functions for consistency. EDIT: These have been split off into separate PRs and merged.

The remainder of the patches then build up a stub implementation of pthreads for THREAD_MODEL=single. I have done my best to try to make sure that all functions are as conforming as possible (under the assumption that another thread cannot ever be launched). This means that objects such as mutexes and rwlocks actually do update their state and will correctly fail when locks cannot be acquired.

When an inevitable deadlock occurs, I have chosen to return EDEADLK when it has been explicitly listed as a permissible return value, and to invoke __builtin_trap otherwise.

I have tested this by rebuilding libc++ with threads enabled and then smoke-testing Clang/LLVM-on-WASI to make sure that it can compile a simple program. I have not run any more-extensive conformance testing.

Fixes #501

yamt · 2024-07-18T03:09:43Z

does this break pthread detection mechanisms like cmake find_package(Threads)?

ArcaneNibble · 2024-07-18T03:31:46Z

does this break pthread detection mechanisms like cmake find_package(Threads)?

I don't know. I don't use cmake beyond building other software that uses it.

Several of the comments in the linked issue are implying that, even if it might, not having these stubs is causing even more difficulty with porting software.

libc-top-half/musl/arch/wasm32/pthread_arch.h

Makefile

libc-top-half/musl/src/thread/pthread_atfork.c

libc-top-half/musl/src/thread/pthread_attr_setguardsize.c

yamt · 2024-08-02T04:27:34Z

does this break pthread detection mechanisms like cmake find_package(Threads)?

I don't know. I don't use cmake beyond building other software that uses it.

Several of the comments in the linked issue are implying that, even if it might, not having these stubs is causing even more difficulty with porting software.

having a stub for mutex etc is probably fine.

otoh, i feel it isn't a great idea to provide pthread_create.

This is one part of breaking up #518 into smaller PRs. This should be (IMO) the least-controversial batch of commits

This is the next part of breaking up #518 into smaller PRs. This is the rest of the commits which change the existing `THREAD_MODEL=posix` functionality. It: * _removes_ some functions which are optional and which were already nonfunctional. * (Continues to) establish a precedent of trying to remain as compatible with Open Group specifications as possible, even when there are major differences in WASI capabilities (i.e. thread cancellation) Compared to the RFC PR, the `pthread_atfork` stub has been dropped as it is now officially obsolete as of the very recent Issue 8 of the specifications.

ArcaneNibble · 2024-08-06T20:35:49Z

does this break pthread detection mechanisms like cmake find_package(Threads)?

I don't know. I don't use cmake beyond building other software that uses it.
Several of the comments in the linked issue are implying that, even if it might, not having these stubs is causing even more difficulty with porting software.

having a stub for mutex etc is probably fine.

otoh, i feel it isn't a great idea to provide pthread_create.

Although there are also thoughts of potentially changing it, libcxx currently does require/expect pthread_create to exist. Incidentally, it performs detection whether or not pthreads is supported by looking for pthread_once and not pthread_create (although said detection doesn't actually seem to work properly for some reason, possibly because wasm handles undefined symbols by importing them from env rather than failing?)

ArcaneNibble · 2024-08-06T20:37:17Z

I've rebased this PR and fixed the issues that CI uncovered. I do wish to merge this, but a decision needs to be made about how to handle the "pthreads detection in existing code" issue.

sbc100 · 2024-08-06T20:55:55Z

libc-top-half/stub-pthreads/spinlock.c

@@ -0,0 +1,21 @@
+


Should we call this directory pthread-stubs?

And maybe it should live at the top level alongside dlmalloc and emmalloc (which override/replace libc functions too). @sunfishcode any preference?

ArcaneNibble · 2024-08-16T17:18:28Z

ping?

yamt · 2024-08-19T08:54:09Z

does this break pthread detection mechanisms like cmake find_package(Threads)?

I don't know. I don't use cmake beyond building other software that uses it.
Several of the comments in the linked issue are implying that, even if it might, not having these stubs is causing even more difficulty with porting software.

having a stub for mutex etc is probably fine.
otoh, i feel it isn't a great idea to provide pthread_create.

Although there are also thoughts of potentially changing it, libcxx currently does require/expect pthread_create to exist. Incidentally, it performs detection whether or not pthreads is supported by looking for pthread_once and not pthread_create (although said detection doesn't actually seem to work properly for some reason, possibly because wasm handles undefined symbols by importing them from env rather than failing?)

anyway, libcxx is not a representative application, i suspect.

can you explain why having a stub pthread_create is important for you?
otherwise, can you consider to drop it?

ArcaneNibble · 2024-08-19T10:36:45Z

The immediate reason is: I want to be able to build clang+LLVM for WASI (using @whitequark's patchset), which requires threading functions inside libcxx. The usage of types such as std::mutex is so pervasive throughout the LLVM codebase that it will probably never be acceptable to put compile-time conditionals around all of them. This is what started the desire for some kind of dummy threading stubs.

As far as I can tell, the only possible choices are:

do what I've done here, where all of the stubs (i.e. enough to build libcxx with threads enabled) exist inside wasi-libc. Subsequently change wasi-sdk such that all builds have threads enabled inside libcxx. Use the resulting wasi-sdk to build clang+LLVM
do choice 1 except introduce three threading models (i.e. "single", "stub", "posix"). This was discussed in a review comment with @sbc100, who didn't seem to see it necessary to do that.
make it entirely libcxx's problem
1. by making libcxx carry stubs for everything (synchronization and nonfunctional thread creation). I predict that this would result in simply having this exact same discussion in a brand-new GitHub thread.
2. by making libcxx carry stubs for synchronization primitives only, still removing e.g. std::thread. This would subsequently require hunting down all uses of these types inside the clang+LLVM codebase in order to build the final intended application, but this looks much more doable compared to trying to remove e.g. std::mutex. There does exist some desire to do this. However, this doesn't handle any C code that would need pthreads stubs, only C++ code. I am currently trying to avoid having to spend the effort required to make this option work.
reject upstreaming a stub for pthread_create here (keeping only synchronization primitive stubs). This would mean that libcxx still wouldn't build.
1. Fix libcxx to support "synchronization primitives exist but pthread_create doesn't" as a platform. I don't know about the libcxx maintainers, but I personally would consider this very inelegant and clearly a hack, as such a "platform" would certainly not be POSIX-conformant. This can technically be done by simply conditionally guarding one line of code inside libcxx in order to end up with a libcxx where std::thread exists but always fails, but once again "I predict that this would result in simply having this exact same discussion in a brand-new GitHub thread." Alternatively, this can be done by introducing the ability for libcxx to have std::thread etc. deleted but std::mutex etc. present. This would be just as difficult as doing option 3.ii.
2. Do option 3.i or 3.ii anyways. This works but results in twice the amount of stubs existing.
3. Some downstream user creates a stub for pthread_create anyways and subsequently rebuilds libcxx against it. This is strictly worse as you're now doing the exact same thing except for each downstream user.

The tl;dr is "I need libcxx. I am trying to avoid having to put in all of the effort needed to enable a new 'libcxx has synchronization primitives but no thread creation' build mode for libcxx. I also generally don't think omitting pthread_create is a particularly good idea."

In general, the code I want to get ported to WASI would look something like

// in do_work_{foo|bar|baz|qux}.c
if (use_multiple_threads) {
  pthread_create(...);
} else {
  do_work_synchronously();
}
// in main.c
if (check_command_line_flags("-threads"))
  use_multiple_threads = 1;

With full pthreads stubs in wasi-libc, this code can either be ported by doing nothing ("just" don't pass the -threads command line argument) or by having a single #ifdef __WASI__ in main.c, instead of multiple #ifdefs sprinkled throughout all of the do_work_*.c parts of the program.

Can you perhaps elaborate further as to why you oppose having pthread_create?

whitequark · 2024-08-19T10:40:06Z

(Thanks for doing this work, @ArcaneNibble)

yamt · 2024-08-20T05:47:35Z

The immediate reason is: I want to be able to build clang+LLVM for WASI (using @whitequark's patchset), which requires threading functions inside libcxx. The usage of types such as std::mutex is so pervasive throughout the LLVM codebase that it will probably never be acceptable to put compile-time conditionals around all of them. This is what started the desire for some kind of dummy threading stubs.

how about having libwasi-emulated-pthread.a to follow the precedents?

Can you perhaps elaborate further as to why you oppose having pthread_create?

i don't want to break pthread-detection mechanisms around there, including cmake FindThreads.

ArcaneNibble · 2024-08-20T11:15:24Z

how about having libwasi-emulated-pthread.a to follow the precedents?

Oh! I quite like this idea! I would still want libcxx to be built with threading support enabled for both single and posix though, and I hope to avoid having to hack up its build system to do something special.

i don't want to break pthread-detection mechanisms around there, including cmake FindThreads.

The deliberate intention was to cause detection mechanisms to now detect "this platform has pthreads." This is why I implemented all of the stub functions s.t. they comply with the POSIX specifications (rather than just returning 0 like some other examples that have been posted in #501 ). EAGAIN is explicitly listed as a permissible way for pthread_create to fail, so software should be expected to do something in response.

Can we get community consensus on the approach of "put single-threaded pthreads stubs into libwasi-emulated-pthread.a but always build libcxx with threading enabled (thus causing a program to have undefined symbols if it uses libcxx threading but doesn't link the .a file)" before I go ahead and implement it?

yamt · 2024-08-21T07:08:07Z

I would still want libcxx to be built with threading support enabled for both single and posix though

do you mean to make wasi-sdk ship libcxx built with LIBCXX_ENABLE_THREADS=ON?

why?

ArcaneNibble · 2024-08-21T08:24:40Z

do you mean to make wasi-sdk ship libcxx built with LIBCXX_ENABLE_THREADS=ON?

why?

Yes, because I specifically want to easily port software which uses C++11 threading. i.e. the exact same reason why I am trying to enable pthreads stubs

ArcaneNibble · 2024-08-21T08:34:33Z

@yamt I feel like we are repeatedly talking past each other. Is there any way we can jump into a real-time discussion (e.g. IRC)?

ArcaneNibble · 2024-09-04T00:55:26Z

ping, is there some way through which we can arrive at community consensus about how this pthreads stub functionality should be named and exposed?

bjorn3 · 2024-09-24T18:57:33Z

libc-top-half/stub-pthreads/stub-pthreads.c

+		{PTHREAD_THREADS_MAX} would be exceeded."
+	*/
+	return EAGAIN;
+}


(I don't have write permission for this repo)

It feels to me like pthread_create, pthread_detach and pthread_join should be kept in a separate archive just like mmap (libwasi-emulated-mman.a) and the signal handler stubs (libwasi-emulated-signal.a) pthread_once and pthread_exit are fine to keep in the main libc.a given that (unlike pthread_create and co) they are actual implementations behaving as they should according to the POSIX specification, albeit only working correctly on single threaded targets, but they are only used on single threaded targets anyway.

I would be happy to test a proposal implementing this as a solution. @sbc100 @yamt Do you think we could use that approach?

ArcaneNibble · 2024-09-25T18:19:03Z

As there have been multiple people in support of something like this, I have:

moved the stub files to a new top-level directory (not inside libc-top-half anymore)
linked the synchronization primitives into libc
put pthread_create, pthread_detach, pthread_join into libwasi-emulated-pthread.a

stub-pthreads/stub-pthreads-emulated.c

sbc100

This approach seems reasonable to me.

stub-pthreads/README.md

abrown

I like this PR and took a look today. I think my main hang-up is the question about when these stubs should (a) fail with a trap, (b) silently do nothing, and (c) work as intended. I thought we'd have a lot of (b) seeing as they're stubs but understand why we occasionally need (a) I guess. I suppose you chose (c) in some cases, like simple constructors, because it was easy enough to add? Or were there some real needs for this in your Clang use case?

abrown · 2024-10-01T23:55:17Z

libc-top-half/musl/src/thread/pthread_self.c

@@ -1,8 +1,7 @@
 #include "pthread_impl.h"
 #include <threads.h>

-#if !defined(__wasilibc_unmodified_upstream) && defined(__wasm__) &&           \
-    defined(_REENTRANT)
+#if !defined(__wasilibc_unmodified_upstream) && defined(__wasm__)


If I'm reading things right, you're keeping the convention of conditionally compiling the threads stuff (stub or otherwise) behind _REENTRANT. Why remove it here? Or, conversely, why add it in the previous file (pthread_impl.h)? One of these seems inconsistent unless I'm missing something.

abrown · 2024-10-01T23:59:23Z

stub-pthreads/barrier.c

+{
+	if (!b->_b_limit) return PTHREAD_BARRIER_SERIAL_THREAD;
+	__builtin_trap();
+}


I'm confused here:

pthread_barrier_init is a copy of the original function, i.e., it works

pthread_barrier_destroy does nothing, but silently

pthread_barrier_wait traps

Shouldn't we more coherent and, e.g., "just trap" for all three or "just work"?

Let's look at it again:

pthread_barrier_init creates a barrier in a way appropriate for a single-threaded environment.

pthread_barrier_destroy destroys a barrier in a way appropriate for a single-threaded environment, which in this case happens to do nothing.
-pthread_barrier_wait waits on a barrier in a way appropriate for a single-threaded environment, i.e. traps since this would result in a deadlock, except for the 1-thread case where it returns PTHREAD_BARRIER_SERIAL_THREAD.

If I didn't miss anything, this is a conformant single-threaded implementation, so there is nothing that "doesn't work" there.

I see what you mean; let me look at this again.

Ok, I was misreading when this stub code was being used: I see now that it is only in single-threaded "stub" scenarios, i.e., #ifndef _REENTRANT.

whitequark · 2024-10-02T23:01:51Z

Or were there some real needs for this in your Clang use case?

Clang does actually need the initialization and destruction of primitives to succeed, this is not just an arbitrary choice. Overall, I like @ArcaneNibble's chosen strategy, which is to produce a conformant single-threaded implementation that traps on deadlock instead of halting.

abrown

I get it, looks good to me. It will be interesting to see how this holds up when we add proper pthread tests. From the poking I did, the implementations included here do make sense under the assumption that this is only run in single-threaded WebAssembly.

abrown · 2024-10-10T01:30:52Z

stub-pthreads/barrier.c

+{
+	if (!b->_b_limit) return PTHREAD_BARRIER_SERIAL_THREAD;
+	__builtin_trap();
+}


Ok, I was misreading when this stub code was being used: I see now that it is only in single-threaded "stub" scenarios, i.e., #ifndef _REENTRANT.

abrown · 2024-10-10T02:58:51Z

stub-pthreads/stub-pthreads-good.c

@@ -0,0 +1,43 @@
+// This file is linked into libc


One thing that led to my earlier confusion is the naming of these files, coupled with the lack of documentation. I think the organization of all of this boils down to:

for non *-threads targets, we compile in the single-threaded stub implementations plus this file, stub-pthreads-good.c

for the libwasi-emulated-pthread.{so|a} libraries, we compile the stub-pthreads-emulated.c file; we expect users to link that library solely with a non *-threads libc

for *-threads targets, none of this code is compiled

I can see now what the intent is here, though. I might be tempted to refactor this a bit in the future but, if this is immediately useful to some you, let's go with it as-is for now. Some of the confusion is also caused by how the Makefile is structured, etc.

abrown · 2024-10-10T03:02:34Z

Oh, one last comment: @ArcaneNibble, as I reviewed this, I noticed that the recent #522 modifies the expected files so we can't merge this as-is. Can you rebase or merge that in?

This is needed in order to *correctly* implement things such as cancellation handlers being invoked on pthread_exit

This fixes compatibility with old versions of clang

This is "good" as it actually does perform all the cancellation actions and then calls exit()

These functions always fail

ArcaneNibble · 2024-10-10T15:59:03Z

Rebased

carlocab · 2024-12-13T14:00:33Z

libc-top-half/musl/include/pthread.h

+#define pthread_create(...) ({ _Static_assert(0, "This mode of WASI does not have threads enabled; \
+to enable stub functions which always fail, \
+compile with -D_WASI_EMULATED_PTHREAD and link with -lwasi-emulated-pthread"); 0;})


Apologies if this is already being discussed elsewhere (wasn't able to find it), but:

We're hitting this assertion in Homebrew even when targeting wasm32-wasip1-threads.¹² This is just from building a simple hello world program in C++.

Is that intentional? If yes, then perhaps the message should omit the part about -lwasi-emulated-pthread, since that should not be necessary when targeting wasm32-wasip1-threads?

Footnotes

https://github.com/Homebrew/homebrew-core/actions/runs/12304619382/job/34345998203#step:3:219 ↩

https://github.com/Homebrew/homebrew-core/actions/runs/12304619382/job/34345998203#step:3:185 ↩

Filed #558 for this.

ArcaneNibble force-pushed the dummy_pthread branch from 3d5abda to a6adc10 Compare July 18, 2024 13:58

sbc100 reviewed Jul 18, 2024

View reviewed changes

ArcaneNibble mentioned this pull request Jul 23, 2024

Improve some pthreads stub functions, batch 0 #519

Merged

sbc100 pushed a commit that referenced this pull request Aug 2, 2024

Improve some pthreads stub functions, batch 0 (#519)

5d3c5e9

This is one part of breaking up #518 into smaller PRs. This should be (IMO) the least-controversial batch of commits

ArcaneNibble mentioned this pull request Aug 6, 2024

Improve some pthreads stub functions, batch 1 #525

Merged

ArcaneNibble changed the title ~~[RFC] Implement a stub pthreads library for THREAD_MODEL=single~~ Implement a stub pthreads library for THREAD_MODEL=single Aug 6, 2024

ArcaneNibble force-pushed the dummy_pthread branch 2 times, most recently from f47ff55 to 80e0c80 Compare August 6, 2024 20:25

sbc100 reviewed Aug 6, 2024

View reviewed changes

bjorn3 reviewed Sep 24, 2024

View reviewed changes

kateinoigakukun mentioned this pull request Sep 25, 2024

ICU-22838 Add WebAssembly/WASI cross-compilation support unicode-org/icu#3067

Open

7 tasks

ArcaneNibble force-pushed the dummy_pthread branch from 80e0c80 to 87d8ee2 Compare September 25, 2024 18:17

bjorn3 reviewed Sep 25, 2024

View reviewed changes

stub-pthreads/stub-pthreads-emulated.c Show resolved Hide resolved

ArcaneNibble requested a review from sbc100 September 25, 2024 19:10

sbc100 approved these changes Sep 25, 2024

View reviewed changes

stub-pthreads/README.md Outdated Show resolved Hide resolved

ArcaneNibble force-pushed the dummy_pthread branch 2 times, most recently from bf6015b to 260e8ce Compare September 25, 2024 20:36

abrown self-assigned this Sep 27, 2024

abrown reviewed Oct 2, 2024

View reviewed changes

abrown approved these changes Oct 10, 2024

View reviewed changes

ArcaneNibble added 13 commits October 10, 2024 16:56

Add dummy implementation of pthread_attr for single-threaded model

50d7d2d

Add a stub implementation of barriers

7e41cef

Add a stub implementation of mutexes

be912c4

Add a stub implementation of condvars

7d6fc8f

Add a stub implementation of rwlocks

ce2d57d

Add a stub implementation of spinlocks

406a6f3

Enable pthread_self

d0feeba

This is needed in order to *correctly* implement things such as cancellation handlers being invoked on pthread_exit

Enable a number of functions that now work now that we have pthread_self

cc0fbfe

Don't invoke notify builtin in single-threaded mode

45adc9f

This fixes compatibility with old versions of clang

Add a stub for pthread_once

7ea3949

Implement single_threaded pthread_exit

93fd9f5

This is "good" as it actually does perform all the cancellation actions and then calls exit()

Add emulated thread-creation-related functions

1b0706d

These functions always fail

Make always-fail functions error if _WASI_EMULATED_PTHREAD isn't defined

9b80a8e

ArcaneNibble force-pushed the dummy_pthread branch from 260e8ce to 9b80a8e Compare October 10, 2024 15:58

abrown merged commit a05277a into WebAssembly:main Oct 10, 2024
8 checks passed

pavelsavara mentioned this pull request Oct 23, 2024

Add 'stream' and 'future' types WebAssembly/component-model#405

Merged

sunfishcode mentioned this pull request Dec 3, 2024

Multi-threading and Atomics WebAssembly/WASI#296

Closed

carlocab reviewed Dec 13, 2024

View reviewed changes

carlocab mentioned this pull request Dec 13, 2024

Thread-related asserts trigger in C++ even when targeting thread-enabled WASI #558

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement a stub pthreads library for `THREAD_MODEL=single` #518

Implement a stub pthreads library for `THREAD_MODEL=single` #518

ArcaneNibble commented Jul 18, 2024 •

edited

Loading

yamt commented Jul 18, 2024

ArcaneNibble commented Jul 18, 2024

yamt commented Aug 2, 2024

ArcaneNibble commented Aug 6, 2024

ArcaneNibble commented Aug 6, 2024

sbc100 Aug 6, 2024

ArcaneNibble commented Aug 16, 2024

yamt commented Aug 19, 2024

ArcaneNibble commented Aug 19, 2024

whitequark commented Aug 19, 2024

yamt commented Aug 20, 2024

ArcaneNibble commented Aug 20, 2024

yamt commented Aug 21, 2024

ArcaneNibble commented Aug 21, 2024

ArcaneNibble commented Aug 21, 2024

ArcaneNibble commented Sep 4, 2024

bjorn3 Sep 24, 2024

whitequark Sep 24, 2024

ArcaneNibble commented Sep 25, 2024

sbc100 left a comment

abrown left a comment

abrown Oct 1, 2024

abrown Oct 1, 2024

whitequark Oct 2, 2024

abrown Oct 5, 2024

abrown Oct 10, 2024

whitequark commented Oct 2, 2024 •

edited

Loading

abrown left a comment

abrown Oct 10, 2024

abrown Oct 10, 2024

abrown commented Oct 10, 2024

ArcaneNibble commented Oct 10, 2024

carlocab Dec 13, 2024 •

edited

Loading

carlocab Dec 13, 2024

Implement a stub pthreads library for THREAD_MODEL=single #518

Implement a stub pthreads library for THREAD_MODEL=single #518

Conversation

ArcaneNibble commented Jul 18, 2024 • edited Loading

yamt commented Jul 18, 2024

ArcaneNibble commented Jul 18, 2024

yamt commented Aug 2, 2024

ArcaneNibble commented Aug 6, 2024

ArcaneNibble commented Aug 6, 2024

Choose a reason for hiding this comment

ArcaneNibble commented Aug 16, 2024

yamt commented Aug 19, 2024

ArcaneNibble commented Aug 19, 2024

whitequark commented Aug 19, 2024

yamt commented Aug 20, 2024

ArcaneNibble commented Aug 20, 2024

yamt commented Aug 21, 2024

ArcaneNibble commented Aug 21, 2024

ArcaneNibble commented Aug 21, 2024

ArcaneNibble commented Sep 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ArcaneNibble commented Sep 25, 2024

sbc100 left a comment

Choose a reason for hiding this comment

abrown left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whitequark commented Oct 2, 2024 • edited Loading

abrown left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abrown commented Oct 10, 2024

ArcaneNibble commented Oct 10, 2024

carlocab Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

Footnotes

Choose a reason for hiding this comment

Implement a stub pthreads library for `THREAD_MODEL=single` #518

Implement a stub pthreads library for `THREAD_MODEL=single` #518

ArcaneNibble commented Jul 18, 2024 •

edited

Loading

whitequark commented Oct 2, 2024 •

edited

Loading

carlocab Dec 13, 2024 •

edited

Loading