Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support pickling tasks #3

Open
2 tasks
alcrene opened this issue Dec 24, 2021 · 0 comments
Open
2 tasks

Support pickling tasks #3

alcrene opened this issue Dec 24, 2021 · 0 comments

Comments

@alcrene
Copy link
Owner

alcrene commented Dec 24, 2021

Although pickling simple minimal tasks sometimes works, this a) is untested and b) fails as soon as inputs or outputs involve dynamically created types (which are quite common in smttask). Thus we need

  • Support for pickling tasks with dynamic types. These may be introduced via
    • task generators (task_generators.py)
    • multi-outputs (see base.TaskOutput.__init_subclass__)
  • Comprehensive tests
    • Include all types of tasks (esp. all generated types) and both single- and multi-output tasks.
    • Test multiprocessing where:
      • Tasks are passed as arguments
      • Tasks are returned as results
      • Tasks are executed in the subprocess, and their result is returned.

The main use case I see for supporting pickle is to allow parallelizing task execution with the multiprocessing module.

Relevant SO question: Pickling dynamically generated classes?

alcrene added a commit that referenced this issue Dec 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant