Locking multiple physics systems from a single thread

[mihe]

What's the policy on interacting with multiple physics systems from the same thread? I noticed while moving some locks around that PhysicsLock checks for deadlocks using a static thread_local, and seems to trigger an assert if you lock multiple physics systems from a single thread, which seems to suggest that users are meant to only have a single system locked per thread.

Is there some inherent limitation with having multiple systems locked at the same time, or is it just the assert/check that's a bit heavy-handed?

It's not a showstopper or anything, but I figured I should check before attempting to work around it.

[jrouwe]

Hello,

That's a good point that I didn't think about! The PhysicsSystems are completely independent of each other as long as you make sure that you don't mix bodies or constraints between them (mixing shapes should be ok). It is totally fine to lock bodies from multiple PhysicsSystems at the same time. All the thread_locals that are used for validating consistency were not made with multiple PhysicsSystems in mind and will falsely assert that you're doing something wrong. I'll have to come up with a better way of tracking the PhysicsLocks (and also BodyAccess, BodyManager::sOverride*).

Note that there are a couple of globals that make the PhysicsSystems not completely independent:

• There is only 1 RTTI factory (Factory::sInstance)
• There is only 1 default material (PhysicsMaterial::sDefault)
• There is only 1 collision dispatcher (CollisionDispatch::sCollideShape / sCastShape)
• ShapeFunctions
• The memalloc, trace and assert functions

But I think these are so low level that you probably wouldn't want different implementations for the different PhysicsSystems.

[mihe]

Thank you!

[mihe]

Any particular reason why this was limited to 4 concurrently locked systems? I was hoping to be able to lock all of Godot's physics spaces/worlds at the same time, of which there can be almost any number. Something like a 4-player split screen where each viewport has its own world (for whatever reason) will exceed this number when running in the editor.

[jrouwe]

I limited it to 4 because I didn't want to do dynamic allocations for thread locals because it is sure to break with a custom memory allocator.

The alternative of having to store which thread is locking which mutex in e.g. the BodyManager was also not really appealing as it would mean a shared data structure, which would need some way of locking, which would probably completely change the characteristics of the body locks.

Also I couldn't think of any reason why you would want to lock more than 2 systems at the same time (1 for reading data, 1 for writing data). What is the exact use case?

I'm wondering: Do you need locking at all for Godot? My guess is that there are never going to be multiple threads calling into the physics system at the same time because Godot's physics system doesn't support this.

[mihe]

Do you need locking at all for Godot?

Not necessarily, no.

I've been sort of "handwavy" about the threading in my extension, with some vague plan about doing something about it further down the line. Everything is currently happening on Godot's main thread, besides what Jolt is offloading internally of course, which is plenty.

The only reason I'm using Jolt's locking interfaces at all was to sort of "keep myself honest" as far as accessing Jolt and not paint myself into some deadlocked corner if I ever wanted to make use of locking later on, with the plan to just switch to the non-locking ones if I decided not to go that route. I realize this probably isn't a good approach for something as pervasive as the threading model of a system, but I knew almost nothing about Godot nor Jolt when I started this, and I don't have a ton of experience with multi-threading in general, so I'm sort of figuring things out as I go along.

What is the exact use case?

To put it differently: I've essentially been using Jolt's locks to synchronize the main thread with some imaginary future set of threads that I might or might not come to use for stepping Jolt's physics systems. This meant that when Godot signals to my physics server (as Godot calls it) that it's time to read/write state, I was planning to lock all bodies of all active physics systems/spaces/worlds in one go, read/write state accordingly and then unlock them.

What I'm realizing as I'm typing this out is that if I were to step Jolt's physics systems on one or more separate threads/workers, assuming that even makes sense to begin with, then I'd still probably need to sync up with those at the above mentioned point anyway, meaning there wouldn't be a need to lock the physics systems in the first place, I guess.

Would you say it's worth bothering with trying to step multiple physics systems in parallel? I assume each system already tries to saturate the cores given to them? Maybe I'm overthinking this.

My guess is that there are never going to be multiple threads calling into the physics system at the same time because Godot's physics system doesn't support this.

This is something that's up to the implementer of the physics server to decide, as I understand it. Users can technically call directly into the physics server singleton from pretty much anywhere, including other threads. The physics server is of course not obligated to support this, but the default physics server does support this (in a way) if you enable a specific project setting, although a subset of the API is still off-limits outside of the sync point linked above.

The way they implement this is by wrapping the physics server in another physics server that runs the original one on a separate thread and which does nothing but proxy calls across the threads. I presume it synchronizes if needed, as there doesn't seem to be any notion of asynchronous fetching of state.

This wrapper/proxy is however not something that's available in the extension API, and while it's certainly possible to implement anyway, I figured there might be some other options available. But given the amount of mutable state I'm currently hoarding I'm not thrilled at the idea of making it thread-safe to access my physics server directly.

Anyway, apologies for the long-winded reply, as well as the misguided initial question. Hopefully something good came out of the changes you made. I clearly need to take a closer look at whether, and if so how, to approach threading.

[jrouwe]

Would you say it's worth bothering with trying to step multiple physics systems in parallel? I assume each system already tries to saturate the cores given to them? Maybe I'm overthinking this.

The system tries to use all cores indeed, but there are always moments where nothing sensible can be done. So I would imagine that when you simulate a whole bunch of physics systems that it could be beneficial to run multiple systems in parallel. The interface Jolt offers is not ideal for that as everything happens inside the Update() call so you can't start all systems from the same thread and then wait for all to finish. I can probably make that a bit easier if needed by splitting the Update in a Start and Finish call. In that case you can (and probably should) share the JobSystem between these updating PhysicsSystems to avoid creating more threads than CPU cores.

I'm trying to make sense of the sync() and end_sync() functions. As far as I can see it's only used for asserting that the physics server is accessible, and as far as I understand the code that should be the case outside the step() function (which waits inside the function for the step to be complete). I think sync() and end_sync() could probably be a no-op for Jolt as long as you use the locking interface whenever you read/write a body through the physics server API. If the PhysicsSystem::Update is running, all bodies will be locked, so it should not be possible to get a concurrency issue there.

The way they implement this is by wrapping the physics server in another physics server that runs the original one on a separate thread and which does nothing but proxy calls across the threads. I presume it synchronizes if needed, as there doesn't seem to be any notion of asynchronous fetching of state.

Ah I see! It looks like all calls are posted on an event queue and then a semaphore is used to wait for the result from the wrapped physics server running on a separate thread. When it signals the result is ready, the call returns with the result. So if you don't use Jolt's locks then that physics_server_3d_mt would still be needed but using Jolts locks would probably be more efficient (if you indeed get your own state to be thread safe).

Anyway, I would focus on making things work before solving all the multithreading issues. All of this is going to add a lot of complexity.

Anyway, apologies for the long-winded reply, as well as the misguided initial question. Hopefully something good came out of the changes you made. I clearly need to take a closer look at whether, and if so how, to approach threading.

Yes, something good came out of it for sure! You found a bug in the system and I learned something new about Godot :)