Make spaCy-nlp functions faster

[jbesomi]

(Edit)

Almost all functions of the nlp module under-the-hoods make use of spaCy.

In general, spaCy is quite fast as it uses Cython.

The core code looks like this:

new_data = []
for row in nlp.pipe(s.values, batch_size=32):
        new_data.append( ... row ...)

spacy pipe has been initially chosen as it's multi-threading. An alternative might be to use apply ( probably is slower).

The pipe functions have among other the n_threads as well as the batch_size arguments. Tuning this values might be very important.

This task consists in:

1. Understand spaCy pipe
2. Test on a large dataset different combinations of n_threads and batch_size value
3. (it it make sense) Compare this results with the pandas apply approach
4. Pick the best solution and implement it in all NLP functions that uses spaCy under-the-hoods

• We might find that the optimal values of n_threads and batch_size are not always the same, in this case, we will need to add it as arguments to the NLP functions and update the docstring.

Useful resources:

Turbo-charge your spaCy NLP pipeline

[mk2510]

Hi Jonny,

we will now start with this issue and implement multi_processing, wherever it is useful. 🚀

[jbesomi]

Hi Max,

I guess this task might take quite long time, what if we prioritize to completely finish part 2 of the "API next checklist" and then move on to part 4?

[henrifroese]

Will do 🥈

[jbesomi]

Dask vs. spaCy

It's faster to use pipe from spaCy or to directly use Dask (Dask Dataframe)?

Dask might be the solution we were looking for ...