-
Notifications
You must be signed in to change notification settings - Fork 137
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FR] concurrent map? #388
Comments
Given that Emacs threads are cooperative, what sort of API/semantics do you envision? |
@basil-conto I am mostly a noob on this stuff, and I can't think how we can make a cooperative thread running synchronous code yield. Python's threads are non-cooperative, no? Considering Python also has a global interpreter lock, why couldn't the devs make the threads non-cooperative? But the primitive I am envisioning does not need to touch the state; So can't we somehow just fork the current process? Sth along these lines: (defun (n) (some-expensive-computation n))
(parallel-map #'f '(1 2 3)) The above code should fork, and run Of course, this is a very "simple" architecture, and using ZeroMQ sockets etc might be more efficient, but I just thought up the simplest thing I could think of. The following API might be even be better: (defun (n) (some-expensive-computation n))
(parallel-map-async #'f '(1 2 3)
(lambda (new-list) ...)) (I.e., emacs doesn't block when polling the file locations, and will run the supplied lambda with the result when it is available.) |
https://elpa.gnu.org/packages/async.html Having said that, what kind of scenario would this be useful in? i.e. what kind of long-running, CPU-intensive activity would benefit from mapping one-process-per-element of a list? |
I have a network heavy function in mind; Essentially, I have a list of IDs, and I want to get some remote JSON object from them. Doing this sequentially takes forever. That async package needs one to load their elisp functions in the subprocesses, which is quite damning, as loading everything takes so much time to make the whole enterprise fruitless, and loading just the elisp one needs, needs heavy refactoring and package management. (It’s not a real fork; It is just starting up new emacs processes from scratch.) |
For that case, there are solutions that don't involve multiple Emacs processes, like asynchronous network requests using FYI, I collect some info about writing Emacs packages here: https://github.com/alphapapa/emacs-package-dev-handbook |
BTW, asynchronous subprocess I/O like that is one of the well-defined times at which Emacs can switch thread context. |
More likely would be one-process-per-several-elements. |
Feel free to close the issue. I like my own "simple" API much better than learning about arcane library-specific APIs though. It's a general primitive, and easy to learn, and doesn't require rewriting other parts of the code. Is it even possible to do it, or will forking emacs mess with, idk, its open file handlers or sth? (Why is that
This can easily become a user supplied variable so it doesn't matter. |
Can we have some functional concurrency primitives, like a concurrent map that runs some function on a sequence concurrently?
Of course, parallelism would be best, but I think emacs threads don't support that?
The text was updated successfully, but these errors were encountered: