Another person stuck trying to embed python (call into python) from threads.

[djacobow]

Hi there. I have an app in C++ that uses a lot of threads, and for testing and rapid development, it would be really nice for them to be able to call into python implementations of various things rather than implementing them all in C++.

I understand that because of the GIL, effective python execution will be through a single thread, or perhaps more accurately, "as if" through a single thread, but I'm still struggling to get it to work at all. What I think I have observed is that, no matter what I do with gil_scoped_acquire and gil_scoped_release, the only thread that can successfully call into python is the one that holds scoped_interpreter.

As a result, I've had to write a serializing wrapper (not shown) that all my threads call into that pushes python function calls and their arguments into a thread-safe queue, and the "python thread" runs a pop-execute loop fires a condition variable to notify the original calling thread of the result. This works, but I feel like it should be unnecessary, and is, strictly speaking, worse than even normal python threads since it cannot release the gil until completion of the function rather than real python that can swap threads when blocked on io.

Is there a better way to do this? I'd like to not have my serializer since I am told that the GIL is going away and I'd prefer that my code continue to be serialized when it does.

Here's a simplified example of what I tried first:

class naive_c {
  public:
    naive_c() : interp_(py::scoped_interpreter()) {
        py::exec("print('constructor')");
    }

    void bang() {
        std::thread t1([]() { py::exec("print('bang t1')"); });
        std::thread t2([]() { py::exec("print('bang t2')"); });
        t1.join();
        t2.join();
    };

  private:
    py::scoped_interpreter interp_;
};

int main(int ac, char *av[]) {
    auto n = std::make_unique<naive_c>();
    n->bang();
};

This crashes. I naively thought py::exec would automatically take/release the GIL as required, but apparently nope.

Then I tried this, and a bunch of variations:

class less_naive_c {
  public:
    less_naive_c() : interp_(py::scoped_interpreter()) {
        py::exec("print('constructor')");
        py::gil_scoped_release r {}; 
    }   

    void bang() {
        std::thread t1([]() {
            py::gil_scoped_acquire a {}; 
            py::exec("print('bang t1')");
        });
        std::thread t2([]() {
            py::gil_scoped_acquire a {}; 
            py::exec("print('bang t2')");
        });
        t1.join();
        t2.join();
    };  

  private:
    py::scoped_interpreter interp_;
};

int main(int ac, char *av[]) {
    auto n = std::make_unique<less_naive_c>();
    n->bang();
};

This does not crash, but it deadlocks.

Finally, the code below crashes, too, but after running the thread. I think what is happening is that in thread t1 the interpreter is created and when it exits it is not destroyed because the object holds it. When code (from the main thread) tries to destroy the object at the end of the program, it crashes because it's not the same thread.

class just_one_c {
  public:
    just_one_c() {}

    void bang() {
        std::thread t1([this]() {
            interp_ = std::make_unique<py::scoped_interpreter>();
            py::exec("print('bang t1')");
        });
        t1.join();
    }

  private:
    std::unique_ptr<py::scoped_interpreter> interp_;
};

I guess the last one is not unexpected.

I'm just curious if there is a better way to do this. Ideally, I'd like to just call into pybind::exec without any extra work, and it would be pybind that waits for the GIL, killing my thread parallelism, but at least the program runs.

Am I missing something?

Using:
Linux Ubuntu 24.04.1
clang 18.1.3
pybind 11 2.13.16
python 3.12.3

Best,
Dave J