Parallel NumPy seamlessly speeds up NumPy for large arrays (64K+ elements) with no change required to your existing NumPy code.
PNumPy supports Linux, Windows, and MacOS for NumPy >= 1.18 for python 3.6, 3.7, 3.8, and 3.9.
This first release speeds up NumPy binary and unary ufuncs such as add, multiply, isnan, abs, sin, log, sum, min and many more. Sped up functions also include: sort, argsort, lexsort, arange, boolean indexing, and fancy indexing. In the near future we will speed up: astype, where, putmask, and searchsorted.
Other packages that use numpy, such as scikit-learn or pandas, will also be sped up for large arrays.
pip install pnumpy
You can also install the latest development versions with
pip install https://github.com/Quansight/pnumpy/archive/main.zip
See the full documentation
To use the project:
import pnumpy as pnParallel NumPy speeds up NumPy silently under the hood. To see some benchmarks yourself run
pn.benchmark()
To get a partial list of functions sped up run
pn.atop_info()
To disable or enable pnumpy run
pn.disable()
pn.enable()
PNumPy provides additional routines such as converting a NumPy record array to a column major array in parallel (pn.recarray_to_colmajor) which is useful for DataFrames. Other routines include pn.lexsort32, which performs an indirect sort using np.int32 instead of np.int64 consuming half the memory and running faster.
PNumPy uses a combination of threads and 256 bit vector intrinsics to speed up calculations. By default most operations will only use 3 additional worker threads in combination with the main python thread for a total 4. Large arrays are divided up into 16K chunks and threads are assigned to maintain cache coherency. More threads are dynamically deployed for more intensive CPU problems like np.sin. Users can customize threading. The example below shows how 4 threads can work together to quadruple the effective L2 cache size.
To cap the number of additional worker threads to 3 run
pn.thread_setworkers(3)
To disable or re-enable threading run
pn.thread_disable()
pn.thread_enable()
To disable or re-enable just the atop engine run
pn.atop_disable()
pn.atop_enable()
Q: If I type np.sort(a) where a is an array, will it be sped up?
A: If len(a) > 65536 and pnumpy has been imported, it will automatically be sped up
Q: How is sort sped up?
A: PNumPy uses additional threads to divide up the sorting job. For example it might perform an 8 way quicksort followed by a 4 way mergesort
Q: How is scikit or pandas sped up?
A: PNumPy's vector loops and threads will speed up any package that uses large NumPy arrays
To run all the tests run:
python -m pip install pytest
python -m pytest tests