Support for multiprocessing engines (polars or moddin pandas) #688
FelixGonzalez42
started this conversation in
Feature Requests
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Mito is an excellent extension to facilitate tasks, however sometimes it falls short in speed, and it's not Mito's fault, it's pandas' fault.
Pandas does not allow multicore processing, which causes most of the hardware to go to waste while a single nuclide is overused, to solve this, the community has created several alternatives to pandas, the first is Moddin, which uses pandas as a base and allows just by changing the pandas import to modin ex (import modin.pandas as pd) use pandas with the multithreading power of dask or ray.
The better alternative is polars, which is built from the ground up in the almighty rust for multithreading, which makes it infinitely faster than pandas and Modin, even faster than cudf which uses graphics card multithreading, would be awesome if Mito would support these engines, since they would improve the processing speed of mito up to 10X.
https://pola-rs.github.io/polars-book/user-guide/coming_from_pandas.html
https://docs.rapids.ai/api/cudf/stable/
https://modin.readthedocs.io/en/latest/
Beta Was this translation helpful? Give feedback.
All reactions