Replace numpy.random.RandomState with SFC64 - for speed #2864

zygm0nt · 2020-06-22T08:58:10Z

This PR is related to #2782

It changes the randomness generator from obsolete RandomState to the fastest alternative currently SFC64.

Closes #2782

piskvorky · 2020-06-22T09:26:29Z

Needs a significant and detailed description (currently empty). Can you please extend it?

What is the motivation?
Who is affected and why? Why this model and not others?
Effects of the PR: benchmarks, potential pitfalls, backward compatibility?

Plus unit tests. Cheers.

zygm0nt · 2020-06-22T12:41:55Z

@piskvorky I've added a short description, because it's just a PR for the mentioned issue.

I've observed the same problem as mentioned in the issue and found the fix helps.

Would gladly create a test for that, but can't think of a way to check a) speed, b) randomness - in such a test. Any suggestions?

piskvorky · 2020-06-22T13:24:00Z

Thanks! Can you include some benchmark numbers, before-after, on a realistic (non-trivial) dataset? E.g. text9 or wikipedia.

gojomo · 2020-06-23T01:17:45Z

This code doesn't yet even allow its single modified method to complete, because the SFC64 instance does not implement the .rand() method called on the next line. All unit tests that use this method are failing.

The SFC64 must be wrapped in a Generator, and then its method random() should be called instead or rand().

But, once that's fixed, just a single before/after comparison on a run that takes a few minutes (text8 or text9) should be enough, to be sure the change is not having the opposite of the intended effect.

It's just a 2-line fix, to match the Numpy docs' performance recommendations, which:

no longer recommend the old MT19973 generator, noting that it both "fails some statistical tests and is not especially fast compared to modern PRNGs".
recommend either PCG64, in general, or SFC64, if we want speed & don't need 'jumpability' – which seems exactly our need.

piskvorky · 2020-06-23T08:34:58Z

Oh wow. @zygm0nt you haven't run this code at all?

zygm0nt · 2020-06-23T11:34:05Z

Sorry for that. Will fix the errors, than let you know.

zygm0nt · 2020-06-23T13:19:25Z

@piskvorky @gojomo I've fixed the issue, now almost all tests are passing. Travis reports errors, but don't see the connection here with changes from this PR.

As advised I'll prepare a test run with standard dataset. How should I provide it? Log from a run would suffice?

gojomo · 2020-06-24T00:49:09Z

The one test that's failing appears unrelated to this change.

(It's a type assertion in test_keras_integration.py, which hasn't changed since 2018, and only failing in the Python 3.8 tests, passing in 3.7 and 3.6... whereas all of 3.8, 3.7, & 3.6 passed on develop just 8 days ago. I wouldn't hold back this change to investigate; if it persists, it can be looked into separately.)

Just ensuring a nontrivial example runs before and after, and checking that the logged timings indicate no worsening of the relevant 'resetting model weights' step (and excerpting that report here or in the issue) would be enough for me, given all other relevant tests passing.

mpenkov · 2020-06-25T02:16:35Z

The one test that's failing appears unrelated to this change.

It appears unrelated, but the same test is passing on the master branch. If we move forward with this PR, we should:

Skip the test under Py3.8 to make sure everything stays green
Open up a separate issue to investigate, fix the underlying issue and re-enable the test

@zygm0nt Please do (1). I'll do (2).

mpenkov · 2020-06-25T02:17:29Z

gensim/models/word2vec.py

@@ -1691,8 +1691,10 @@ def prepare_weights(self, hs, negative, wv, update=False, vocabulary=None):
    def seeded_vector(self, seed_string, vector_size):
        """Get a random vector (but deterministic by seed_string)."""
        # Note: built-in hash() may vary by Python version or even (in Py3.x) per launch
-        once = random.RandomState(self.hashfxn(seed_string) & 0xffffffff)
-        return (once.rand(vector_size) - 0.5) / vector_size
+        once = random.Generator(


This can all fit on one line, surely?

gojomo · 2020-06-25T05:40:39Z

Why would we want 'everything green' by checking in version-specific test-skips - cruft that we'll have to remember to undo later?

If it's a legit test that failing, it should keep failing until it's fixed. If the changes here didn't cause the failure – but rather revealed something else that broke elsewhere recently – that shouldn't hold up this fix.

(I'm not also sure it's accurate to say everything's currently "passing" on the master branch - only that it passed on the last commit ~9 days ago. A simple retry of the build might reveal the same problem there.)

mpenkov · 2020-06-25T06:28:22Z

Why would we want 'everything green' by checking in version-specific test-skips - cruft that we'll have to remember to undo later?

Because that's how we know at a glance that everything still works. Without the all-green, we will have to check the failed CI runs for each build (did it fail because we expected it to fail, or was there another, unexpected problem?).

We won't have to remember to remove anything. We'll open a ticket for it and deal with it when we have the bandwidth.

If it's a legit test that failing, it should keep failing until it's fixed.

First, we don't know if the test is failing because there's a problem with the code, its dependencies, or the test itself (or some combination of the above).

Second, if keep failing tests around, they'll fail for every build until we fix them. This means that for every build, we'll have to dig through CI logs and determine if the failure is expected or not. This is significant overhead for a maintainer. I'm strongly against this - I'd rather spend my time more productively, plus there's no guarantee that manual checks are 100% correct - we'll always get the call wrong once in a while.

If the changes here didn't cause the failure – but rather revealed something else that broke elsewhere recently – that shouldn't hold up this fix.

There are two ways forward:

Merge this PR and deal with the problem later
Deal with the problem first and then merge this PR

We're not in a terrible rush to merge this, so my preference is (2). The amount of effort required for dealing with the problem in the immediate sense is low (disabling a test is easy), anyway.

gojomo · 2020-06-25T18:03:28Z

But why would we want to "know at a glance that everything still works" when in fact, everything still doesn't work – a test is broken. Special-casing a test to be off for one Python version (if in fact that's even the real pattern-of-failure here – unclear from a single observation) just creates a dishonest green-light. It's like removing the battery from a smoke-detector instead of investigating "what's that burning smell?"

And it contaminates this PR/commit with a totally unrelated bit of kludge code that's destined to be removed ASAP.

Again, it's quite possible the trunk is already broken by the same mysterious (or intermittent) factor that caused this. (If you have a button that tries a rebuild, please try it. This discussion about protecting the green may be moot.) Every attempt to build on that should see this problem until fixed, and thus I do agree with your (2) "Deal with the problem first and then merge this PR" - to the extent that I wouldn't want anyone to do any temporary test-disablements until the problem is better understood.

mpenkov · 2020-06-26T03:41:58Z

@gojomo Let's continue our discussion in #2866.

zygm0nt · 2020-06-26T12:22:05Z

Didn't have time to setup the environment with mentioned datasets (wikipedia, text8). But I've repeated the training with my target dataset. And must say that this change resulted in dramatic reduce in time of resetting weights.

First log is from original training:

2020-06-21 21:53:43,103 [12113] INFO     gensim.models.word2vec: Loading a fresh vocabulary
2020-06-21 21:55:30,397 [12113] INFO     gensim.models.word2vec: effective_min_count=3 retains 19115982 unique words (53% of original 35448710, drops 16332728)
2020-06-21 21:55:30,397 [12113] INFO     gensim.models.word2vec: effective_min_count=3 leaves 950533618 word corpus (97% of original 972371594, drops 21837976)
2020-06-21 21:56:38,747 [12113] INFO     gensim.models.word2vec: deleting the raw counts dictionary of 35448710 items
2020-06-21 21:56:39,623 [12113] INFO     gensim.models.word2vec: sample=1e-05 downsamples 3133 most-common words
2020-06-21 21:56:39,623 [12113] INFO     gensim.models.word2vec: downsampling leaves estimated 641405759 word corpus (67.5% of prior 950533618)
2020-06-21 21:58:03,009 [12113] INFO     gensim.models.base_any2vec: estimated required memory for 19115982 words and 100 dimensions: 24850776600 bytes
2020-06-21 21:58:03,010 [12113] INFO     gensim.models.word2vec: resetting layer weights
2020-06-21 23:04:38,790 [12113] INFO     gensim.models.base_any2vec: training model with 64 workers on 19115982 vocabulary and 100 features, using sg=1 hs=0 sample=1e-05 negative=5 window=15

The current newest version (3.8.3) takes ~1h15 minutes to reset weights.

This log is with the fixed version:

2020-06-26 11:53:44,527 [4680] INFO     gensim.models.word2vec: Loading a fresh vocabulary
2020-06-26 11:55:34,153 [4680] INFO     gensim.models.word2vec: effective_min_count=3 retains 19115982 unique words (53% of original 35448710, drops 16332728)
2020-06-26 11:55:34,153 [4680] INFO     gensim.models.word2vec: effective_min_count=3 leaves 950533618 word corpus (97% of original 972371594, drops 21837976)
2020-06-26 11:56:44,745 [4680] INFO     gensim.models.word2vec: deleting the raw counts dictionary of 35448710 items
2020-06-26 11:56:45,636 [4680] INFO     gensim.models.word2vec: sample=1e-05 downsamples 3133 most-common words
2020-06-26 11:56:45,636 [4680] INFO     gensim.models.word2vec: downsampling leaves estimated 641405759 word corpus (67.5% of prior 950533618)
2020-06-26 11:58:12,694 [4680] INFO     gensim.models.base_any2vec: estimated required memory for 19115982 words and 100 dimensions: 24850776600 bytes
2020-06-26 11:58:12,694 [4680] INFO     gensim.models.word2vec: resetting layer weights
2020-06-26 12:13:09,172 [4680] INFO     gensim.models.base_any2vec: training model with 64 workers on 19115982 vocabulary and 100 features, using sg=1 hs=0 sample=1e-05 negative=5 window=15

So, the time difference between resetting layer weights log and the next log is ~14minutes, with fix.

Would you like me to present also the results for wikipedia/text8?

gojomo · 2020-06-26T15:59:40Z

Personally, I think this is enough to apply the quick fix: a 5x speedup, and matching the numpy recommendations.

Stylistically I'd agree with @mpenkov's suggestion that this Generator+algorithm instantiation can fit on 1-line. (That seems the idiomatic usage in numpy example code, too.)

It is likely we could get a significant further speedup by letting numpy fill the whole array in one call, but we'd lose the current slight nod to determinism, where each vector's initial value is seeded by its key. (Thus, subsequent runs with the same model seed – but even with different vocabularies via a changed min_count etc – start each word at the same random place. Yet, given all the other things pulling against reproducible runs, especially multithreading, & my belief users should generally be discouraged from seeking reproducibility with such stochastic algorithms, this seeding trick probably isn't worth the performance cost.) But that'd be for a future PR.

mpenkov · 2020-06-27T13:03:11Z

@zygm0nt Please merge upstream/develop to fix the problem with keras.

zygm0nt · 2020-06-29T06:19:46Z

Done, rebased and fixed stylistic remarks.

piskvorky · 2020-06-29T08:39:07Z

Probably too late now, but avoid force-pushing to shared branches. It makes reviewing harder and loses history.

mpenkov · 2020-06-29T11:55:57Z

@zygm0nt We've merged this. Thank you for your contribution!

zygm0nt · 2020-06-29T14:22:52Z

Thanks!

@TSV

* added release/check_wheels.py (#2610) * added release/check_wheels.py * added preamble * Update release/check_wheels.py Co-Authored-By: Radim Řehůřek <me@radimrehurek.com> * respond to review comments * Add hacktoberfest-related documentation (#2616) * git add HACKTOBERFEST.md * clarify contributions * respond to review comments * add link to HACKTOBERFEST.md from README.md * typo * include comments from Gordon * Fixed #2554 (#2619) * Properly install Pattern library for documentation build (#2626) * Probably fixes #2534 * Uppercase P * Added comment * Disable Py2.7 builds under Travis, CircleCI and AppVeyor (#2601) * Disable Py2.7 builds under Travis and AppVeyor * use Py3.7.4 image under CircleCI * tweak circleci config.yml * patch tox.ini * more fixes to get docs building under tox * s/python3.7/python3/ * delay annoy ImportError until actual use * bring back Pattern * simplify invokation of pip command * add install_numpy_scipy.py * fixup * use sys.executable * adjust version in install_wheels.py * adjust travis.yml * adjust version in install_wheels.py back * add logging statements * use version_info instead of sys.version * fixup * Handling for iterables without 0-th element, fixes #2556 (#2629) * Handling for iterables without 0-th element, fixes #2556 * Improved accessing the first element for the case of big datasets * Move Py2 deprecation warning to top of changelog (#2627) It belongs at the top. People should see it immediately without having to scroll down to an older release. * Change find_interlinks return type to list of tuples (#2636) * Change interlinks format to list of tuples. Fixes #2635 This commit fixes the issue in #2635 This commit changes the interlinks storage in the `segment_wiki` script from dictionary to a list of tuples. We can process the test wikidata used in the test suite of gensim to inspect the new behavior. ``` python gensim/scripts/segment_wiki.py -i \ -f ~/Downloads/enwiki-latest-pages-articles1.xml-p000000010p000030302-shortened.bz2 \ -o ~/Downloads/enwiki-latest.json.gz ``` We get the following output: ``` $ cat ~/Downloads/enwiki-latest.json.gz | zcat | head -1 | jq -r '.interlinks[] | [.[0], .[1]] | @TSV' | sort | head -ism -ism 1848 Revolution 1848 Revolution 1917 October Revolution 1917 October Revolution 6 February 1934 crisis February 1934 riots A. S. Neill A. S. Neill AK Press AK Press Abu Hanifa Abu Hanifa Adolf Brand Adolf Brand Adolf Brand Adolf Brand Adolf Hitler Hitler ``` All tests pass for the related test file. ``` python -m unittest gensim.test.test_scripts /Users/smishra/miniconda3/envs/TwitterNER/lib/python3.7/bz2.py:131: ResourceWarning: unclosed file <_io.BufferedReader name='/Users/smishra/workspace/codes/python/gensim/gensim/test/test_data/enwiki-latest-pages-articles1.xml-p000000010p000030302-shortened.bz2'> self._buffer = None ResourceWarning: Enable tracemalloc to get the object allocation traceback ..... ---------------------------------------------------------------------- Ran 5 tests in 6.298s OK ``` * Updated docstrings * Fixed flake8 issue of long line in docsrtring * Fixed comments and replaces assertTrue with assertEqual * Fixed unittest comment and checks for wikicorpus * Improve gensim documentation (numfocus) (#2591) * Update makefile to point to new subdirectory * Update layout.html to show new documentation sections * introduce sphinx gallery * reorganize gallery * trim tut3.rst * git add docs/to_python.py * git add gallery/010_tutorials/run_doc2vec_lee.py * minor layout tweak * add downloader api howto * add fasttext tutorial and howto * use pprint in fasttext tutorial * add summarization tutorial * git add gallery/020_howtos/run_howto_compare_lda.py * add fasttext thumbnails * adding core concepts tutorial * add summarization plot * update notebook to use 20newsgroups * update notebook * improve notebook * update howtos * fix distance metrics tutorial * improve distance_metrics.ipynb * git add gallery/010_tutorials/run_distance_metrics.py * git add gallery/020_howtos/run_news_classification.py * move downloader API to tutorials section * add docs/src/auto_examples so bindr can pick up the notebooks * minor changes * git add gallery/010_tutorials/run_lda.py * more minor changes * More minor changes * git add gallery/010_tutorials/run_word2vec.py * updated notebooks * git add gallery/010_tutorials/run_wmd.py * add image * move parts of intro.rst to core concepts tutorial * move README.txt to wiki * get rid of fasttext wrapper tutorial * update top-level heading * more minor changes * minor updates * improve Doc2Vec tutorial, move explanations from IMDB * git add gallery/020_howtos/run_doc2vec_imdb.py * git st * fix notebook paths for bindr * rename gallery to documentation * git add binder/requirements.txt * git add auto_examples/000_core/requirements.txt * adding requirements.txt for binder * removing requirements files added in desperation * update conf.py * remove temporary files from git branch * rm images * merge "getting started" into "core concepts" * add some clarifying text * add Jupyter notebook * Revert "get rid of fasttext wrapper tutorial" This reverts commit 3ec0a46. * get rid of fasttext wrapper guide * git add auto_examples/ * minor fixes * fix typo * add listing of corpora and models * get rid of binder * git add gallery/020_howtos/run_doc.py * more instructions for authorship * improve linkage between core tutorials * add highlighting * move downloader to howto * restore support and about sections * sync toolbars * Add installation instructions to top page * clean up html * add wordcloud-based thumbnails * updated notebooks * update script * add sphinx-gallery to doc dependencies * include memory_profiler in docs_testenv * git add README.rst * use proper temporary file * reorganize tutorials section * clarify version control in README.rst * git rm 020_howtos/saved_model_wrapper * move pivoted document normalization to tutorials section * fix ordering in howto section * add images * add annoy to doc dependencies * update gitignore * disable tox spinner * turn off progress bar for pip * fix labels * naming fixes * git rm docs/notebooks/gensim\ Quick\ Start.ipynb * git rm docs/notebooks/Corpora_and_Vector_Spaces.ipynb * git rm gensim\ Quick\ Start.ipynb * git rm docs/notebooks/Topics_and_Transformations.ipynb * git rm docs/notebooks/Similarity_Queries.ipynb * git rm docs/notebooks/summarization_tutorial.ipynb * git rm docs/notebooks/distance_metrics.ipynb * git rm docs/notebooks/word2vec.ipynb * git rm docs/notebooks/doc2vec-lee.ipynb * git rm docs/notebooks/gensim_news_classification.ipynb * git rm docs/notebooks/lda_training_tips.ipynb * git rm docs/notebooks/doc2vec-IMDB.ipynb * git rm docs/notebooks/annoytutorial.ipynb * git rm tutorial.rst tut1.rst tut2.rst tut3.rst * minor update to layout.html * git rm changes_080.rst * minor tweaks to gallery and surrounding docs * remove cruft from run_doc2vec_imdb.py * update doc howto * fixup * git add requirements_docs.txt * more dependencies in requirements_docs.txt * re-enable LDA howto * add missing images * add built LDA howto * port tutorials.md to gallery * WIP: cleaning up docs * language clean up + pin exact versions in doc requirements * git add redirects.csv test_redirects.py * remove gensim_numfocus namespace qualifier * doc cleanup in Other resources * fix redirects * regenerated tutorials * Added tools/check_gallery.py * committing unsuccessful attempt to fix a tutorial before deleting it * remove tutorials that don't work * index page fixes * add install anchor * Update redirects.csv * link fixes from local testing * replace easy_install with pip * renamed run_040_compare_lda.py to run_compare_lda.py * minor fixes * more fixes from website testing * updating wordcloud images * add pandas to requirements_docs.txt * !! * more dependency + code fixes * update upload path to "live" website * update test_redirects.py * git rm redirects.csv test_redirects.py * fix setup.py to get documentation to build under CircleCI (#2650) * Fix links to documentation in README.md (#2646) * Fix links to documentation in README.md * Update README.md * Delete requirements.txt (#2648) * Remove native Python implementations of Cython extensions (#2630) * Remove native Python implementations of Cython extensions Fix #2511 * remove print statement in tox.ini * remove print statement in tox.ini * fix flake8 issues * fix missing imports * adjust exception message * bring back FAST_VERSION variable * fixup: missing parens * disable progress bar for tox * respond to review comments * remove C/C++ sources generated from Cython files * update setup.py * remove duplicate line in setup.py * fix numpy bootstrapping * update tox.ini * handle cython dependency in setup.py * fixup in setup.py: lowercase c * more cython sourcery * fix tox.ini * Fix merge artifact in setup.py * fix merge artifact * disable pip progress bar under CircleCI * replacing deleted notebooks with placeholders (#2654) * Document accessing model's vocabulary (#2661) * document accessing model's vocabulary * update images * Improve explanation of top_chain_var parameter in Dynamic Topic Model (DTM) documentation * improve & corrected gensim documentation (#2637) * more descriptive explanation of top_chain_var * Comment out Hacktober Fest from README (#2677) - uncomment next year * Update word2vec2tensor.py (#2678) * Speed up word2vec model loading (#2671) * Speed up word2vec binary model loading (#2642) * Add correctness tests for optimized word2vec model loading (#2642) * Include remarks of Radim to code speeding up vectors loading (#2671) * Include remarks of Michael to code speeding up vectors loading (#2671) * Refactor _load_word2vec_format into a few functions for better readability * Clean-up _add_word_to_result function * Fix local import degrading the performance of word2vec model loading (#2671) (#2682) * [Issue-2670] Bug fix: Initialize doc_no2 because it is not set when corpus' is empty (#2672) * [Issue-2670] Bug fix: Initialize doc_no2 because it is not set when 'corpus' is empty * [Issue-2670] Add: unittests should fail on invalid input (generator and empty corpus) * [Issue-2670] Add: Fix unittest for generator * [Issue-2670] Fix unittest tox:flake8 errors * [Issue-2670] Fix: empty corpus def in unittest * [Issue-2670] Fix: empty corpus and generator unittests * [Issue-2670] Fix: empty corpus and generator unittests * Warn when BM25.average_idf < 0 (#2687) Closes #2684 * Rerun Soft Cosine Measure tutorial notebook (#2691) * Fix simple typo: voacab -> vocab (#2719) Closes #2718 * Fix appveyor builds (#2706) * move install_wheels script * git add continuous_integration/check_wheels.py * bump versions for numpy and scipy * update old requirements.txt * add file header * get rid of install_wheels.py hack * fixup: update travis.yml * Update continuous_integration/check_wheels.py Co-Authored-By: Radim Řehůřek <me@radimrehurek.com> * Update continuous_integration/check_wheels.py Co-Authored-By: Radim Řehůřek <me@radimrehurek.com> Co-authored-by: Radim Řehůřek <me@radimrehurek.com> * Change similarity strategy when finding n best (#2720) * Find largest by absolute value * Add helper function to simplify code & add unit test for it * Initialize self.cfs in Dictionary.compatify method (#2618) * Fix for #2574 * Fix for #2574 * Fix ValueError when instantiating SparseTermSimilarityMatrix (#2689) * force python int before calling islice. islice don't accept numpy int * add test to check islice error * it makes test to fail * make sure that islice receives a python int * fix typo * Refactor bm25 to include model parametrization (cont.) (#2722) * Refactor bm25 to include model parametrization * Refactor constants back and fix typo * Refactor parameters order and description * Add BM25 tests This closes #2597 and closes #2606 * Simplify asserts in BM25 tests * Refactor BM25.get_score Co-authored-by: Marcelo d'Almeida <md@id.uff.br> * Fix overflow error for `*Vec` corpusfile-based training (#2700) * long long types for expected_examples & total_documents * regenerate .cpp files * Implement saving to Facebook format (#2712) * Add writing header for binary FB format (#2611) * Adding writing vocabulary, vectors, output layer for FB format (#2611) * Clean up writing to binary FB format (#2611) * Adding tests for saving FastText models to binary FB format (#2611) * Extending tests for saving FastText models to binary FB format (#2611) * Clean up (flake8) writing to binary FB format (#2611) * Word count bug fix + including additional test (#2611) * Removing f-strings for Python 3.5 compatibility + clean-up(#2611) * Clean up the comments (#2611) * Removing forgotten f-string for Python 3.5 compatibility (#2611) * Correct tests failing @ CI (#2611) * Another attempt to correct tests failing @ CI (#2611) * Yet another attempt to correct tests failing @ CI (#2611) * New attempt to correct tests failing @ CI (#2611) * Fix accidentally broken test (#2611) * Include Radim remarks to saving models in binary FB format (#2611) * Correcting loss bug (#2611) * Completed correcting loss bug (#2611) * Correcting breaking doc building bug (#2611) * Include first batch of Michael remarks * Refactoring SaveFacebookFormatRoundtripModelToModelTest according to Michael remarks (#2611) * Refactoring remaining tests according to Michael remarks (#2611) * Cleaning up the test refactoring (#2611) * Refactoring handling tuple result from struct.unpack (#2611) * Removing unused import (#2611) * Refactoring variable name according to Michael review (#2611) * Removing redundant saving in test for Facebook binary saving (#2611) * Minimizing context manager blocks span (#2611) * Remove obsolete comment (#2611) * Shortening method name (#2611) * Moving model parameters to _check_roundtrip function (#2611) * Finished moving model parameters to _check_roundtrip function (#2611) * Clean-up FT_HOME behaviour (#2611) * Simplifying vectors equality check (#2611) * Unifying testing method names (#2611) * Refactoring _create_and_save_fb_model method name (#2611) * Refactoring test names (#2611) * Refactoring flake8 errors (#2611) * Correcting fasttext invocation handling (#2611) * Removing _parse_wordvectors function (#2611) * Correcting whitespace and simplifying test assertion (#2611) * Removing redundant anonymous variable (#2611) * Moving assertion outside of a context manager (#2611) * Function rename (#2611) * Cleaning doc strings and comments in FB binary format saving functionality (#2611) * Cleaning doc strings in end-user API for FB binary format saving (#2611) * Correcting FT_CMD execution in SaveFacebookByteIdentityTest (#2611) * Use time.time instead of time.clock in gensim/models/hdpmodel.py (#2730) * Use time.process_time() instead of time.clock() * time.process_time() -> time.time() * better replacement of deprecated .clock() * drop py35, add py38 (travis), update explicit dependency versions * better CI logs w/ gdb after core dump * improved comments via piskvorky review Co-Authored-By: Radim Řehůřek <me@radimrehurek.com> * rm autogenerated *.cpp files that shouldn't be in source control * Fix TypeError when using the -m flag (#2734) Currently, if you attempt to use the script with the --min-article-character you get an error because it gets parsed a string and the functions expect an int. This fix addresses the issue. ``` Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.6/dist-packages/gensim/scripts/segment_wiki.py", line 385, in <module> include_interlinks=args.include_interlinks File "/usr/local/lib/python3.6/dist-packages/gensim/scripts/segment_wiki.py", line 141, in segment_and_write_all_articles for idx, article in enumerate(article_stream): File "/usr/local/lib/python3.6/dist-packages/gensim/scripts/segment_wiki.py", line 100, in segment_all_articles for article in wiki_sections_text: File "/usr/local/lib/python3.6/dist-packages/gensim/scripts/segment_wiki.py", line 332, in get_texts_with_sections if sum(len(body.strip()) for (_, body) in sections) < self.min_article_character: TypeError: '<' not supported between instances of 'int' and 'str'``` * del cython.sh * Improve documentation in run_similarity_queries example (#2770) * Fix fastText word_vec() for OOV words with use_norm=True (#2764) * add a test for oov similarity * fix a test for oov similarity * fix it once more * prepare the real fix * remove a redundant variable * less accurate comparison Co-authored-by: David Dale <ddale@yandex-team.ru> * remove mention of py27 (#2751) on 25 oct 2019, setup.py was updated to require python 3.5. this change removes the suggestion of testing against py27. * Fix KeyedVectors.add matrix type (#2761) * add type test * cast internal state to passed type * ekv -> kv * parametrize datatype & cast embeddings passed to `add` to KV datatype * set f32 as default type Co-authored-by: Ivan Menshikh <imenshikh@embedika.ru> Co-authored-by: Michael Penkov <m@penkov.dev> * use collections.abc for Mapping (#2750) * use collections.abc.Mapping when available * ignore py2, tox -e py27-linux revealed setup.py requires python 3.5 * use collections.abc.Iterable * Fix out of range issue in gensim.summarization.keywords (#2738) * Fixed out of range error in keywords.py * Now using min() function to improve readability * Added a test to make sure that keywords does not fail when words param is greater than number of words in string * Fixing travisCI build error from not having 2 lines after class definition * Fixed whitespace issue for flake8 Co-authored-by: Carter Olsen <olsencar@oregonstate.edu> * fixed get_keras_embedding, now accepts word mapping (#2676) * fixed get_keras_embedding, now accepts word mapping * skip tests if keras not installed * removed unnessecary comment from test_keyed_vectors * fixed indentation * fixed flake import error * moved skip test decorator to class * Update gensim/models/keyedvectors.py Co-Authored-By: Michael Penkov <m@penkov.dev> * Update gensim/models/keyedvectors.py Co-Authored-By: Michael Penkov <m@penkov.dev> * Update gensim/models/keyedvectors.py Co-Authored-By: Michael Penkov <m@penkov.dev> * renamed keras_installed flag to upper case, removed unneeded comment Co-authored-by: Zhicharevich <Alex_Zhicharevich@intuit.com> Co-authored-by: Michael Penkov <m@penkov.dev> * Add downloads badge to README - idea from piskvorky/smart_open#440 * Get rid of "wheels" badge * link downloads badge to pepy instead of pypi * fix broken english in tests (#2773) * fix build, use KeyedVectors class (#2774) * cElementTree has been deprecated since Python 3.3 and removed in Python 3.9. * Fix FastText RAM usage in tests (+ fixes for wheel building) (#2791) * pin `bucket` parameter (to avoid RAM issues on CI system) + get rid win32 skip * fix flake8 * partially fix doc building * better workaround for docs build * fix sphinx-gallery * avoid test error * get back loading of old model (because large buckets) * Update setup.py Co-Authored-By: Radim Řehůřek <me@radimrehurek.com> * Update gensim/test/test_fasttext.py Co-Authored-By: Radim Řehůřek <me@radimrehurek.com> * define missing buckets & fix formatting Co-authored-by: Ivan Menshikh <imenshikh@embedika.ru> Co-authored-by: Radim Řehůřek <me@radimrehurek.com> * Fix typo in comments\nThe rows of the corpus are actually documents, fix the comment to reduce confusion * Add osx+py38 case for avoid multiprocessing issue (#2800) * add osx+py38 case for avoid multiprocessing issue * add comment, fix warning * extend comment Co-Authored-By: Radim Řehůřek <me@radimrehurek.com> * Update gensim/utils.py * Update gensim/utils.py Co-Authored-By: Michael Penkov <m@penkov.dev> Co-authored-by: Radim Řehůřek <me@radimrehurek.com> Co-authored-by: Michael Penkov <m@penkov.dev> * Use nicer twitter badge * Use downloads badge from shields.io * Use blue in badges * Remove conda-forge badge * Make twitter badge blue, too * Cache badges - use google's caching proxy for img.shields.io badges - fixes #2805 * Use HTML comments instead of Markdown comment - simpler & easier to read and maintain * [MRG] Update README instructions + clean up testing (#2814) * update README instructions * WIP: enable test deps * unpin old tensorflow in tests - old versions not present in newer Pythons * looking into segfault in py3.6 - https://travis-ci.org/github/RaRe-Technologies/gensim/jobs/681096362 * put back pyemd * put keras back * put back tensorflow * investigate segfault in py3.6 * address review comments * avoid py3.6 segfault in Travis tests * Add basic yml file for setup pipeline (will fail) * revert back travis * Replace AppVeyor by Azure Pipelines (#2824) * dummy change of pipeline file * good bye appveyor * no specific trigger * attempt to trigger by PR * ??? * [REVERT ME] Specify only tests that fails * get platform & stay only single test * try to debug * fail fast * no re-runs * meh * hack? * Revert "good bye appveyor" (FOR COMPARISON WITH AZURE) This reverts commit cd57175. * try to understand, where \r comes from * continue debug * raise * upd * okay, try to avoid CRLF * revert back * bye again, appveyor * delete appveyor stuff too * move visdom to linux-only env (to avoid frequent failures on win) * fix docs building * Update CHANGELOG.md (#2829) * Update CHANGELOG.md (#2831) * Fix-2253: Remove docker folder since it fails to build (#2833) * Removed Docker from gensim since docker image fails to build and there's nobody to maintain docker * Remove irrelevant comment about docker * LdaModel documentation update -remove claim that it accepts CSC matrix as input (#2832) * Update LDA model documentation to remove the claim that LDA accepts CSC matrices as an input * Update CHANGELOG.md Co-authored-by: Michael Penkov <m@penkov.dev> * delete .gitattributes (#2836) * delete .gitattributes * disable certain tests on Azure pipelines * tweak env var behavior * disable one more test * make the newest version of flake8 happy * patch tox.ini to pin flake8 and flake8-rst versions Co-authored-by: Michael Penkov <m@penkov.dev> * Fix for Python 3.9/3.10: remove xml.etree.cElementTree (#2846) * Update classifiers in setup.py * Python 3.3+ uses a fast implementation whenever available * Don't import ElementTree as ET * Correct grammar in docs (#2573) * Correct grammar in docs * Update gensim/scripts/glove2word2vec.py Co-Authored-By: Radim Řehůřek <me@radimrehurek.com> * Update gensim/scripts/glove2word2vec.py Co-Authored-By: Michael Penkov <m@penkov.dev> * Update glove2word2vec.py * Update glove2word2vec.py * Update gensim/scripts/glove2word2vec.py Co-authored-by: Radim Řehůřek <me@radimrehurek.com> Co-authored-by: Michael Penkov <m@penkov.dev> * Don't proxy-cache badges with Google Images (#2854) * pin keras=2.3.1 because 2.4.3 causes KerasWord2VecWrappper test failure in Py 3.8 (#2868) work around for #2865 * Expose max_final_vocab parameter in FastText constructor (#2867) * Expose max_final_vocab parameter in FastText constructor * Fix lint error * respond to reviewer comments * add unit test Co-authored-by: Cristi Burca <mail@scribu.net> * Replace numpy.random.RandomState with SFC64 - for speed (#2864) Co-authored-by: Marcin Cylke <marcin.cylke@allegro.pl> * Update CHANGELOG.md * Clarify that license is LGPL-2.1 (#2871) Use only the license attribute with an SPDX license id for V2.1 of the LGPL. The setup.py classifier was inconsistent with the license attribute as one pointed to LGPL-2.0-or-later and one to LGPL-2.1-only which seems to be otherwise the correct license based on header comments and documentation. Classifiers do not support the LGPL v2.1 and are eventually being subsumed by the license field [1] [1] https://discuss.python.org/t/improving-license-clarity-with-better-package-metadata/2154 Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com> * Fix travis issues for latest keras versions. (#2869) * Fix travis issues for latest keras versions. * Unpin keras version. * verify "inputs" and "outputs" named params on keras 2.3.1 * Unpin keras version * Put cell outputs back to the soft cosine measure benchmark notebook (#2808) Revert bcee414 * KeyedVectors & *2Vec API streamlining, consistency (#2698) * slim low-value warnings * clarify vectors/vectors_vocab relationship; fix lockf & nonsense ngram-norming confusion * mv FT, KV tests to right place * rm deprecations, obsolete refs/tests, delete_temporary_training_data, update usages * update usages, tests, flake8 cleanup * expand KeyedVectors to obviate Doc2VecKeyedVectors; upconvert old offset-style doctags * fix docstring warnings; update usages * rm unused old plain-python codepaths * unify class comments under __init__ for consistncy w/ api doc presentation * name/comment harmonization (rm 'entity', lessen 'word'-centricity) * table formatting * return pyemd to linux test env * split backcompat tests for better resolution * convert Vocab & related data items to use dataclasses * rm obsolete Vocab/Trainable/abstract/Wrapper classes, persistent callbacks (bug #2136), outdated tests/warnings; update usages * tune tests for stability, runtimes; rm auto reruns that hide flakiness * fix numpy FutureWarning: arrays to stack must be sequence * (commented-out) deoptimization option * stronger FB model testing; no _unpack_copy test * merge redundant methods; rm duplicated imports/defs * rationalize _lockf, buckets_word behaviors * rename .docvecs to .dv * update usages; rm obsolete tests; restore gensim.utils import * intensify FT tests (more epochs, more buckets) * flake8-3.8.0 style fixes - but also pin flake8-3.7.9 vs 3.8.0 'output_file' error * replace vectors_norm with 1d norms * tighten testParallel * rm .vocab & 'Vocab' classes; add expandable 'vecattrs' * update usages (no vocabs) * enable running inside '-m mtprof' (or cProfile) via explicit unittest.main(module=..) * faster sample_int reads * load_word2vec_format(.., no_header=True) to support GLoVe text vectors * refactor & comment lockf feature; allow single-element lockf * improve FT comment * rm deprecated/unneded init_sims calls * fixes to code style * flake8: fix overlong lines * rm stray merge error * rm duplicated , old nonstandard hash workarounds * use numpy-recommended PRNG constructor * add sg to FastTextConfig & consult it; rm remaining broken-hash cruft * reorg conditional packages for clarity * comments, names, refactoring, randomization * Apply suggestions from code review Co-authored-by: Radim Řehůřek <me@radimrehurek.com> * fix cruft left from suggestion * fix numpy-32bit-on-Windows; executable docs * mv lee_corpus to utils; cleanup * update poincare for latest KV __init__ signature * restore word_vec method for proper overriding, but rm usages * Apply suggestions from code review Co-authored-by: Radim Řehůřek <me@radimrehurek.com> * adjust testParallel against failure risk * intensify training for an occasionally failing test * clarify word/char ngrams handling; rm outdated comments * mostly avoid duplciating FastTextConfig fields into locals * avoid copies/pointers for no-bucket (FT as W2V) case * rm obsolete test (already skipped & somewhat originally misguided) * simpler/faster .get(..., default) (avoids exception-catching in has_index_for) * add default option to get_index; avoid exception in has_index_for * chained range check Co-authored-by: Radim Řehůřek <me@radimrehurek.com> * Update CHANGELOG.md Co-authored-by: Radim Řehůřek <radimrehurek@seznam.cz> Co-authored-by: Radim Řehůřek <me@radimrehurek.com> Co-authored-by: Michael Penkov <m@penkov.dev> * Delete .gitattributes * test showing FT failure as W2V * set .vectors even when ngrams off * use _save_specials/_load_specials per type * Make docs clearer on `alpha` parameter in LDA model * Update Hoffman paper link * rm whitespace * Update gensim/models/ldamodel.py * Update gensim/models/ldamodel.py * Update gensim/models/ldamodel.py * re-applying changes from #2821 * migrating + regenerating changed docs * fix forgotten iteritems * remove extra `model.wv` * split overlong doc line * get rid of six in doc2vec * increase test timeout for Visdom server * add 32/64 bits report * add deprecations for init_sims() * remove vectors_norm + add link to migration guide to deprecation warnings * rename vectors_norm everywhere, update tests, regen docs * put back no-op property setter of deprecated vectors_norm * fix typo * fix flake8 * disable Keras tests - failing with weird errors on py3.7+3.8, see https://travis-ci.org/github/RaRe-Technologies/gensim/jobs/713448950#L862 * test showing FT failure as W2V * set .vectors even when ngrams off * Update gensim/test/test_fasttext.py * Update gensim/test/test_fasttext.py * refresh docs for run_annoy tutorial * Reduce memory use of the term similarity matrix constructor, deprecate the positive_definite parameter, and extend normalization capabilities of the inner_product method (#2783) * Deprecate SparseTermSimilarityMatrix's positive_definite parameter * Reference paper on efficient implementation of soft cosine similarity * Add example with Annoy indexer to SparseTermSimilarityMatrix * Add example of obtaining word embeddings from SparseTermSimilarityMatrix * Reduce space complexity of SparseTermSimilarityMatrix construction Build matrix using arrays and bitfields rather than DOK sparse format This work is based on the following blog post by @maciejkula: https://maciejkula.github.io/2015/02/22/incremental-construction-of-sparse-matrices/ * Fix a typo in the soft cosine similarity Jupyter notebook * Add human-readable string representation for TermSimilarityIndex * Avoid sparse term similarity matrix computation when nonzero_limit <= 0 * Extend normalization in the inner_product method Support the `maintain` vector normalization scheme. Support separate vector normalization schemes for queries and documents. * Remove a note in the docstring of SparseTermSimilarityMatrix * Rerun continuous integration tests * Use ==/!= to compare constant literals * Add human-readable string representation for TermSimilarityIndex (cont.) * Prod flake8 with a coding style violation in a docstring * Collapse two lambdas into one internal function * Revert "Prod flake8 with a coding style violation in a docstring" This reverts commit 6557b84. * Avoid str.format() * Slice SparseTermSimilarityMatrix.inner_product tests by input types * Remove similarity_type_code local variable * Remove starting underscore from local function name * Save indentation level and define populate_buffers function * Extract SparseTermSimilarityMatrix constructor body to _create_source * Extract NON_NEGATIVE_NORM_ASSERTION_MESSAGE to a module-level constant * Extract cell assignment logic to cell_full local function * Split variable swapping into three separate statements * Extract normalization from the body of SparseTermSimilarityMatrix.inner_product * Wrap overlong line * Add test_inner_product_zerovector_zerovector and test_inner_product_zerovector_vector tests * Further split test_inner_product into 63 test cases * Raise ValueError when dictionary is empty * Fix doc2vec crash for large sets of doc-vectors (#2907) * Fix AttributeError in WikiCorpus (#2901) * bug fix: wikicorpus getstream from data file-path \n Replace fname with input * refactor: use property decorator for input Co-authored-by: jshah02 <jenisnehal.shah@factset.com> * Corrected info about elements of the job queue * Add unused args of `_update_alpha` * intensify cbow+hs tests; bulk testing method * use increment operator Co-authored-by: Radim Řehůřek <me@radimrehurek.com> * Change num_words to topn in dtm_coherence (#2926) * Integrate what is essentially the same process * docstirng fixes * get rid of python2 constructs * Remove Keras dependency (#2937) * remove keras dependency - relegate keras exports to FAQ: https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ#q13-how-do-i-export-a-trained-word2vec-model-to-keras * remove forgotten notebook with keras * Update CHANGELOG.md Co-authored-by: Michael Penkov <m@penkov.dev> * code style fixes while debugging pickle model sizes * py2 to 3: get rid of forgotten range * fix docs * get rid of numpy.str_ * Fix deprecations in SoftCosineSimilarity (#2940) * Remove deprecated Soft Cosine Measure parameters, functions, and tests. Here is a detailed list of the deprecations: - Parameter `positive_definite` of `SparseTermSimilarityMatrix` has been renamed to `dominant`. Test `test_positive_definite` has been removed. - Parameter `similarity_matrix` of `SoftCosineSimilarity` no longer accepts unencapsulated sparse matrices. - Parameter `normalized` of `SparseTermSimilarityMatrix.inner_product` no longer accepts booleans. - Function `matutils.softcossim` has been superseded by method `SparseTermSimilarityMatrix.inner_product`. Tests in `TestSoftCosineSimilarity` have been removed. * Remove unused imports * Fix additional warnings from the CI test suite * Update CHANGELOG.md Co-authored-by: Michael Penkov <m@penkov.dev> * Fix "generator" language in word2vec docs (#2935) * Fix docs about Word2Vec (fix #2934) Docs say you can use a generator as the first argument, but you can't. The tempfile path was also unused, so that's been removed. * Fix langauge to make it clear streaming is supported Technically a generator is a kind of iterator, so this clarifies that a restartable iterator (as opposed to a consumable generator) is necessary. * Update gensim/models/word2vec.py * Update CHANGELOG.md Co-authored-by: Michael Penkov <m@penkov.dev> * Bump minimum Python version to 3.6 (#2947) * remove claims of Python 3.5 support brings `setup.py` up to sync with #2713 & #2715 changes * remove py2.7 and py3.5 from web index page * Update CHANGELOG.md Co-authored-by: Radim Řehůřek <radimrehurek@seznam.cz> Co-authored-by: Michael Penkov <m@penkov.dev> * fix index2entity, fix docs, hard-fail deprecated properties * fix typos + more doc fixes + fix failing tests * more index2word => index_to_key fixes * finish method renaming - add() => add_vectors() - add_one() => add_vector() * Update gensim/models/word2vec.py Co-authored-by: Michael Penkov <m@penkov.dev> * a few more style fixes * fix nonsensical word2vec path examples * more doc fixes * `it` => `itertools`, + code style fixes * Refactor ldamulticore to serialize less data (#2300) * fix * Update CHANGELOG.md Co-authored-by: Michael Penkov <m@penkov.dev> * new docs theme * redo copy on web index page * fix docs in KeyedVectors * clean up docs structure * hopepage header update, social panel and new favicon * fix flake8 * reduce space under code section * fix images in core tutorials * WIP: migrating tutorials to 4.0 * fix doc2vec tutorial FIXMEs * add autogenerated docs * fixing flake8 errors * remove gensim.summarization subpackage, docs and test data (#2958) * remove gensim.summarization subpackage, docs and test data * Update changelog * remove old import * Remove distance metrics and pivoted normalization tutorials * reuse from test.utils * test re-saving-native-FT after update-vocab (#2853) * avoid buggy shared list use (#2943) * pre-assert save_facebook_model anomaly * unittest.skipIf instead of pytest.skipIf * refactor init/update vectors/vectors_vocab; bulk randomization * unify/correct Word2Vec & FastText corpus/train parameter checking * suggestions from code review Co-authored-by: Radim Řehůřek <me@radimrehurek.com> * improve train() corpus_iterable parameter doc-comment * disable pytest-rerunfailures due to pytest-dev/pytest-rerunfailures#128 * comment clarity from review * specify dtype to avoid interim float64 * use inefficient-but-all-tests-pass 'uniform' for now, w/ big FIXME comment * refactor phrases * float32 random; diversified dv seed; disable bad test * double-backticks Co-authored-by: Michael Penkov <m@penkov.dev> * inline seed diversifier; unittest.skip * fix phrases tests * clean up rendered docs for phrases * fix sklearn_api.phrases tests + docs - removed testing of loading of old models for backward compatibility, because the wrappers use plain pickle and so don't support SaveLoad overrides * fix flake8 warnings in docstrings * rename export_phrases to find_phrases + add actual export_phrases * skip common english words by default in phrases * sphinx doesn't allow custom section titles :( * use FIXME for comments/doc-comments/names that must change pre-4.0.0 * ignore conjunctions in phrases * make ENGLISH_COMMON_TERMS optional * fix typo * docs: use full version as the "short version" * phrases: rename common_terms => connector_words * fix typo * ReST does not support nested markup * make flake8 shut up * improve HTML doc formatting for consecutive paragraphs * fix typos * add benchmark script * silence flake8 * remove dependency on `six` * regen tutorials * Notification at the top of page in documentation * Update notification.html * Update changelog for 4.0.0 release (#2981) * update changelog for 4.0.0 release * fixup * wip: cleaning up changelog * extend + clean up changelog * note the removal of deprecations in CHANGELOG * finish CHANGELOG - except removed modules, pending info from @mpenkov * CHANGELOG formatting fixes * fix outdated docs - found while updating the migration guide * update migration hyperlinks * fixing fixable FIXMEs, in preparation for 4.0.0beta * fixing iter + size in docstrings * fix typo * clean up logic & docs for KeyedVectors.save_word2vec_format * flake8 fix * py3k: `class X(object):` -> `class X:` * work around issues with flake8-rst * add issues without a PR * improve changelog script * simplify pagination * more flake8-rst fixing Co-authored-by: Radim Řehůřek <radimrehurek@seznam.cz> * bumped version to 4.0.0beta * remove reference to cython.sh * update link in readme * clean up merge artifact Co-authored-by: SanthoshBala18 <santhoshbala18@gmail.com> Co-authored-by: Kirill Malev <playittodeath@gmail.com> Co-authored-by: Shubhanshu Mishra <shubhanshumishra@gmail.com> Co-authored-by: Joel Ong <ong.joel.94@gmail.com> Co-authored-by: Radim Řehůřek <radimrehurek@seznam.cz> Co-authored-by: Karthik Ravi <hikarthikravi@gmail.com> Co-authored-by: lopusz <lopusz@users.noreply.github.com> Co-authored-by: Paul Rigor <paulrigor@users.noreply.github.com> Co-authored-by: Vít Novotný <witiko@mail.muni.cz> Co-authored-by: Tim Gates <tim.gates@iress.com> Co-authored-by: Radim Řehůřek <me@radimrehurek.com> Co-authored-by: Rubanenko Evgeny <erubanenko@gmail.com> Co-authored-by: Pablo Torres <pablo.torres.t@gmail.com> Co-authored-by: Marcelo d'Almeida <md@id.uff.br> Co-authored-by: Dmitry Persiyanov <persiyanov@phystech.edu> Co-authored-by: Wataru Hirota <nobuyoshi2426@gmail.com> Co-authored-by: Gordon Mohr <gojogit@gmail.com> Co-authored-by: Tenoke <sviltodorov@gmail.com> Co-authored-by: Martino Mensio <martinomensio@outlook.it> Co-authored-by: David Dale <dale.david@mail.ru> Co-authored-by: David Dale <ddale@yandex-team.ru> Co-authored-by: Matthew Farrellee <matt@cs.wisc.edu> Co-authored-by: Ivan Menshikh <menshikh.iv@gmail.com> Co-authored-by: Ivan Menshikh <imenshikh@embedika.ru> Co-authored-by: carterols <38794079+carterols@users.noreply.github.com> Co-authored-by: Carter Olsen <olsencar@oregonstate.edu> Co-authored-by: Hamekoded <alex.zhicharevich@gmail.com> Co-authored-by: Zhicharevich <Alex_Zhicharevich@intuit.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Chenxin-Guo <cg633@cornell.edu> Co-authored-by: Faiyaz Hasan <faiyaz.hasan1@gmail.com> Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com> Co-authored-by: Shiv Dhar <shivdhar@gmail.com> Co-authored-by: Cristi Burca <mail@scribu.net> Co-authored-by: Marcin Cylke <marcin.cylke+github@gmail.com> Co-authored-by: Marcin Cylke <marcin.cylke@allegro.pl> Co-authored-by: Philippe Ombredanne <pombredanne@nexb.com> Co-authored-by: Devi Sandeep <sandeep0138@gmail.com> Co-authored-by: S Mono <10430241+xh2@users.noreply.github.com> Co-authored-by: jeni Shah <jenishah@users.noreply.github.com> Co-authored-by: jshah02 <jenisnehal.shah@factset.com> Co-authored-by: lunastera <lounastera@gmail.com> Co-authored-by: Megan <megan.stodel@bbc.co.uk> Co-authored-by: Paul O'Leary McCann <polm@dampfkraft.com> Co-authored-by: horpto <__Singleton__@hackerdom.ru> Co-authored-by: Vaclav Dvorak <admin@wdv.cz>

zygm0nt force-pushed the replace-randomState-with-SFC64 branch from 3f41fd7 to 32c2095 Compare June 23, 2020 12:40

mpenkov reviewed Jun 25, 2020

View reviewed changes

mpenkov mentioned this pull request Jun 26, 2020

temporarily disable failing unit test #2866

Closed

Replace numpy.random.RandomState with SFC64 - for speed

1ef5b06

zygm0nt force-pushed the replace-randomState-with-SFC64 branch from 32c2095 to 1ef5b06 Compare June 29, 2020 06:18

mpenkov merged commit c888b7a into piskvorky:develop Jun 29, 2020

zygm0nt deleted the replace-randomState-with-SFC64 branch June 29, 2020 14:22

piskvorky mentioned this pull request Sep 13, 2021

Resolve NumPy compatibility hell #3231

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace numpy.random.RandomState with SFC64 - for speed #2864

Replace numpy.random.RandomState with SFC64 - for speed #2864

zygm0nt commented Jun 22, 2020 •

edited

Loading

piskvorky commented Jun 22, 2020 •

edited

Loading

zygm0nt commented Jun 22, 2020

piskvorky commented Jun 22, 2020 •

edited

Loading

gojomo commented Jun 23, 2020

piskvorky commented Jun 23, 2020

zygm0nt commented Jun 23, 2020

zygm0nt commented Jun 23, 2020

gojomo commented Jun 24, 2020

mpenkov commented Jun 25, 2020

mpenkov Jun 25, 2020

gojomo commented Jun 25, 2020

mpenkov commented Jun 25, 2020

gojomo commented Jun 25, 2020

mpenkov commented Jun 26, 2020

zygm0nt commented Jun 26, 2020 •

edited

Loading

gojomo commented Jun 26, 2020 •

edited

Loading

mpenkov commented Jun 27, 2020

zygm0nt commented Jun 29, 2020

piskvorky commented Jun 29, 2020

mpenkov commented Jun 29, 2020

zygm0nt commented Jun 29, 2020

Replace numpy.random.RandomState with SFC64 - for speed #2864

Replace numpy.random.RandomState with SFC64 - for speed #2864

Conversation

zygm0nt commented Jun 22, 2020 • edited Loading

piskvorky commented Jun 22, 2020 • edited Loading

zygm0nt commented Jun 22, 2020

piskvorky commented Jun 22, 2020 • edited Loading

gojomo commented Jun 23, 2020

piskvorky commented Jun 23, 2020

zygm0nt commented Jun 23, 2020

zygm0nt commented Jun 23, 2020

gojomo commented Jun 24, 2020

mpenkov commented Jun 25, 2020

mpenkov Jun 25, 2020

Choose a reason for hiding this comment

gojomo commented Jun 25, 2020

mpenkov commented Jun 25, 2020

gojomo commented Jun 25, 2020

mpenkov commented Jun 26, 2020

zygm0nt commented Jun 26, 2020 • edited Loading

gojomo commented Jun 26, 2020 • edited Loading

mpenkov commented Jun 27, 2020

zygm0nt commented Jun 29, 2020

piskvorky commented Jun 29, 2020

mpenkov commented Jun 29, 2020

zygm0nt commented Jun 29, 2020

zygm0nt commented Jun 22, 2020 •

edited

Loading

piskvorky commented Jun 22, 2020 •

edited

Loading

piskvorky commented Jun 22, 2020 •

edited

Loading

zygm0nt commented Jun 26, 2020 •

edited

Loading

gojomo commented Jun 26, 2020 •

edited

Loading