Pin down more specific versions of dependencies

[osma]

I'm using an Annif installation (on the annif-kk machine) that was set up some months ago. When I use tfidf models I get errors like this:

/srv/Annif/.venv/lib/python3.6/site-packages/sklearn/externals/joblib/__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
/srv/Annif/.venv/lib/python3.6/site-packages/sklearn/base.py:306: UserWarning: Trying to unpickle estimator TfidfTransformer from version 0.20.0 when using version 0.21.2. This might lead to breaking code or invalid results. Use at your own risk.
/srv/Annif/.venv/lib/python3.6/site-packages/sklearn/base.py:306: UserWarning: Trying to unpickle estimator TfidfVectorizer from version 0.20.0 when using version 0.21.2. This might lead to breaking code or invalid results. Use at your own risk.

The problem here is that the model (including transformer and vectorizer) was built and saved using an older version of sklearn. Similar problems could happen when gensim, fasttext or vw are updated, because old serialized models may not work properly with the new version.

I think we should pin down the versions of dependencies a bit more precisely. I don't want to depend on specific versions (at least not yet, Annif is still in 0.x) but I think that for the libraries whose data structures are being serialized to disk we should be a bit more conservative, otherwise models may stop working unexpectedly. In the current situation, I think it would make sense to

• pin sklearn to 0.21.x
• pin fasttext(mirror) to the current available minor version (see Upgrade fastText back to official PyPI version #290)
• pin gensim to 3.7.x
• pin vowpalwabbit probably to 8.5.x 8.7.x

[osma]

In future releases, we could then do a more controlled upgrade to newer versions of these dependencies.

[juhoinkinen]

Vowpal Wabbit version 8.7.0.post1 was released 12.7.2019 in Pypi. Should that be used instead of 8.5? The new version needs some new dependencies that are included in #296

[osma]

Yes, we should use VW 8.7 if there are no known issues with that version.
The point of pinning versions is just that dependency upgrades should not come as surprises that are beyond our control and may break things when they happen.

[juhoinkinen]

gensim version 3.8.0 was released 9.7.2019 in Pypi, to which version pinning is set (in branch https://github.com/NatLibFi/Annif/tree/issue291-pin-down-more-specific-versions-of-dependencies)