Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel execution #46

Open
booniepepper opened this issue Jul 23, 2023 · 1 comment
Open

Parallel execution #46

booniepepper opened this issue Jul 23, 2023 · 1 comment
Labels
idea Ideas to ponder
Milestone

Comments

@booniepepper
Copy link
Collaborator

Probably this should wait for async but if I'm feeling cheeky I could write something roughly compatible with it. See https://ziglang.org/news/0.11.0-postponed-again/

Start with a form of each that parallelizes whatever it's doing.

Move on to map and filter and others if it's simple to extend. In anything parallel, ordering of operations, or the resulting data, is not guaranteed. In these cases, there will still need to be locking on the state for any writes.

@booniepepper booniepepper added the idea Ideas to ponder label Jul 24, 2023
@booniepepper
Copy link
Collaborator Author

This really absolutely requires that at least everything within dt's state is following immutable semantics.

This may require reference counting and/or copy-on-write. For example, reversing/sorting a quote/string should have the effect that it only reverses/sorts that single string, and can conceptually be considered as creating a new string. If it's a reverse/sort on the only remaining copy of a quote/string then it's fine if the implementation involves reusing the allocated memory (for example: in-place reverse or sort with no extra allocation)

There can't be a compromise on this from my perspective, it's key to making sure a dt program can be read and remain simple enough to follow exactly what it will perform. (And factor it and refactor it etc)

For side effects... You're on your own! Behavior once you start doing parallel writes to files or sockets or anything else outside the world of dt values will be completely nondeterministic. As an example, a flurry of activity all trying to write or append to the same file will be delegated to whatever filesystem locking behavior exists (if any)

@booniepepper booniepepper added this to the 2.0 milestone Jul 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
idea Ideas to ponder
Projects
None yet
Development

No branches or pull requests

1 participant