Code

Fix critical issues in FastText#2313

Code

Merged

menshikh-iv merged 136 commits intopiskvorky:developpiskvorky/gensim:developfrom mpenkov:attrsCopy head branch name to clipboardJan 11, 2019

+8,951-1,564

Commits

Commits on Dec 16, 2018

• WIP

  mpenkov

  committed
  94a20e9

  Copy full SHA for 94a20e9

• Handle incompatible float size condition

  mpenkov

  committed
  fb2b5b0

  Copy full SHA for fb2b5b0

• update docstring

  mpenkov

  committed
  1a41182

  Copy full SHA for 1a41182

• move regression test to unit tests

  mpenkov

  committed
  cd0b318

  Copy full SHA for cd0b318

Commits on Dec 23, 2018

• WIP

  mpenkov

  committed
  3b31288

  Copy full SHA for 3b31288

• introduced Tracker class

  mpenkov

  committed
  42626a2

  Copy full SHA for 42626a2

• added log examples

  mpenkov

  committed
  12cc3e2

  Copy full SHA for 12cc3e2

Commits on Dec 28, 2018

• initialize trainables weights when loading native model

  mpenkov

  committed
  64f7f39

  Copy full SHA for 64f7f39

• adding script to trigger bug

  mpenkov

  committed
  00b472b

  Copy full SHA for 00b472b

Commits on Dec 29, 2018

• minor documentation changes

  mpenkov

  committed
  abfd573

  Copy full SHA for abfd573

• improve unit test

  mpenkov

  committed
  4e46062

  Copy full SHA for 4e46062

• retrained toy model

  Show description for d3544c7
  mpenkov

  committed
  d3544c7

  Copy full SHA for d3544c7

• update bucket parameter in unit test

  mpenkov

  committed
  b98bc0b

  Copy full SHA for b98bc0b

• update unit test

  mpenkov

  committed
  30be5bd

  Copy full SHA for 30be5bd

• WIP

  mpenkov

  committed
  ab1eaf6

  Copy full SHA for ab1eaf6

Commits on Dec 30, 2018

• retrain model with a smaller dimensionality

  Show description for e59d1db
  mpenkov

  committed
  e59d1db

  Copy full SHA for e59d1db

Commits on Dec 31, 2018

• git add docs/fasttext-notes.md

  mpenkov

  committed
  392201b

  Copy full SHA for 392201b

• adding some comments and fixmes

  mpenkov

  committed
  f25607f

  Copy full SHA for f25607f

• minor refactoring, update tests

  mpenkov

  committed
  ef394ed

  Copy full SHA for ef394ed

• update notes

  mpenkov

  committed
  fe10ca7

  Copy full SHA for fe10ca7

Commits on Jan 2, 2019

• update notes

  mpenkov

  committed
  4c2223c

  Copy full SHA for 4c2223c

• initialize wv.vectors_vocab

  mpenkov

  committed
  28bf757

  Copy full SHA for 28bf757

• init vectors_vocab properly

  mpenkov

  committed
  8e0d04f

  Copy full SHA for 8e0d04f

• add test_sanity_vectors

  mpenkov

  committed
  795fed0

  Copy full SHA for 795fed0

• no longer segfaulting

  mpenkov

  committed
  671b3c0

  Copy full SHA for 671b3c0

• adding tests for in-vocab out-of-vocab words

  mpenkov

  committed
  9adb532

  Copy full SHA for 9adb532

• removing old test

  Show description for 6de08de
  mpenkov

  committed
  6de08de

  Copy full SHA for 6de08de

• fix typo in test, reduce tolerance

  mpenkov

  committed
  81dd478

  Copy full SHA for 81dd478

• update test_continuation, it now fails

  mpenkov

  committed
  5c500f0

  Copy full SHA for 5c500f0

• test continued training with gensim model

  mpenkov

  committed
  cb045de

  Copy full SHA for cb045de

• compare vectors_ngrams before and after

  mpenkov

  committed
  a916266

  Copy full SHA for a916266

• disable test reruns for now

  mpenkov

  committed
  cee6311

  Copy full SHA for cee6311

• set min_count=0

  mpenkov

  committed
  752cf9b

  Copy full SHA for 752cf9b

• initialize wv.buckets_word prior to continuing training

  Show description for ad3342a
  mpenkov

  committed
  ad3342a

  Copy full SHA for ad3342a

• making all tests pass

  mpenkov

  committed
  64caa3c

  Copy full SHA for 64caa3c

• add bucket param to FastTextKeyedVectors constructor

  mpenkov

  committed
  2c9f2b4

  Copy full SHA for 2c9f2b4

• minor refactoring: split out _load_vocab function

  mpenkov

  committed
  80c8092

  Copy full SHA for 80c8092

• minor refactoring: split out _load_trainables method

  mpenkov

  committed
  0d30cae

  Copy full SHA for 0d30cae

Commits on Jan 3, 2019

• removing Tracker class: it was for debugging only

  mpenkov

  committed
  bf1c8b8

  Copy full SHA for bf1c8b8

• remove debugging print statement

  mpenkov

  committed
  ec92983

  Copy full SHA for ec92983

• docstring fixes

  mpenkov

  committed
  1c58119

  Copy full SHA for 1c58119

• remove FIXME, leave this function alone

  mpenkov

  committed
  91b3599

  Copy full SHA for 91b3599

• add newlines at the end of docstrings

  mpenkov

  committed
  5100335

  Copy full SHA for 5100335

• remove comment

  mpenkov

  committed
  aae713d

  Copy full SHA for aae713d

• re-enable test reruns in tox.ini

  mpenkov

  committed
  e5ec723

  Copy full SHA for e5ec723

• remove print statements from tests

  mpenkov

  committed
  87f655a

  Copy full SHA for 87f655a

• git rm trigger.py

  mpenkov

  committed
  76aca9a

  Copy full SHA for 76aca9a

Commits on Jan 4, 2019

• refactor FB model loading code

  Show description for 8027459
  mpenkov

  committed
  8027459

  Copy full SHA for 8027459

• fix bug with missing ngrams (still need cleanup of hash2index & testing)

  menshikh-iv

  committed
  07f34e2

  Copy full SHA for 07f34e2

• fix cython implementation of _ft_hash (based on #2233)

  menshikh-iv

  committed
  118cd7f

  Copy full SHA for 118cd7f

Commits on Jan 5, 2019

• decrease tolerances in unit tests

  mpenkov

  committed
  ef58c7c

  Copy full SHA for ef58c7c

• add test case for native models and hashes

  mpenkov

  committed
  799596d

  Copy full SHA for 799596d

• add working/broken hash implementations for py/cy and tests

  mpenkov

  committed
  6cf3d1f

  Copy full SHA for 6cf3d1f

• minor fixup around hashes

  mpenkov

  committed
  b58a50b

  Copy full SHA for b58a50b

• add oov test

  mpenkov

  committed
  97baf3c

  Copy full SHA for 97baf3c

• adding hash compatibility tests for FastText model

  mpenkov

  committed
  ef90436

  Copy full SHA for ef90436

Commits on Jan 6, 2019

• git rm gensim.xml native.xml

  mpenkov

  committed
  8956530

  Copy full SHA for 8956530

• minor fix in comment

  mpenkov

  committed
  2e10ece

  Copy full SHA for 2e10ece

Commits on Jan 7, 2019

• refactoring: extract _pad_random and _pad ones functions

  mpenkov

  committed
  cb25448

  Copy full SHA for cb25448

• refactoring: move matrix init to FastTextKeyedVectors

  mpenkov

  committed
  901eaeb

  Copy full SHA for 901eaeb

• refactoring: move init_ngrams_post_load method to FastTextKeyedVectors

  mpenkov

  committed
  f0bd22d

  Copy full SHA for f0bd22d

• refactoring: move trainables.get_vocab_word_vecs to wv.calculate_vectors

  mpenkov

  committed
  fa34d84

  Copy full SHA for fa34d84

• refactoring: simplify reset_ngrams_weights method

  mpenkov

  committed
  f9c1547

  Copy full SHA for f9c1547

• deprecate struct_unpack public method

  mpenkov

  committed
  de7d9ef

  Copy full SHA for de7d9ef

• refactoring: improve separation of concerns between model and vectors

  mpenkov

  committed
  2946896

  Copy full SHA for 2946896

• refactoring: improve separation of concerns between model and vectors

  mpenkov

  committed
  a7c14d0

  Copy full SHA for a7c14d0

• refactoring: get rid of init_ngrams_weights method

  mpenkov

  committed
  5598e19

  Copy full SHA for 5598e19

• refactoring: remove unused vectors_vocab_norm attribute

  mpenkov

  committed
  07c84f5

  Copy full SHA for 07c84f5

• review response: update ft_hash_broken comment

  mpenkov

  committed
  f15094d

  Copy full SHA for f15094d

• review response: revert changes to broken hash function

  mpenkov

  committed
  5e25a4f

  Copy full SHA for 5e25a4f

• Merge remote-tracking branch 'upstream/develop' into attrs

  mpenkov

  committed
  b789971

  Copy full SHA for b789971

• review response: handle .bucket backwards compatibility

  mpenkov

  committed
  1ed35ea

  Copy full SHA for 1ed35ea

• review response: adjust warning text

  mpenkov

  committed
  0f62660

  Copy full SHA for 0f62660

• tox -e flake8

  mpenkov

  committed
  c461193

  Copy full SHA for c461193

• tox -e flake8-docs

  mpenkov

  committed
  eeafdec

  Copy full SHA for eeafdec

• review response: store .compatible_hash in vectors only

  mpenkov

  committed
  3e0e656

  Copy full SHA for 3e0e656

• Revert "refactoring: remove unused vectors_vocab_norm attribute"

  Show description for 262599d
  mpenkov

  committed
  262599d

  Copy full SHA for 262599d

• review response: remove critical log comments

  mpenkov

  committed
  7d4e60e

  Copy full SHA for 7d4e60e

• Merge remote-tracking branch 'upstream/develop' into attrs

  mpenkov

  committed
  6cc80de

  Copy full SHA for 6cc80de

• review response: fix docstring in fasttext_bin.py

  Show description for 069912f
  mpenkov

  committed
  069912f

  Copy full SHA for 069912f

• review response: make fasttext_bin an internal module

  mpenkov

  committed
  cc19393

  Copy full SHA for cc19393

• review response: skip cython tests if cython is disabled

  mpenkov

  committed
  1661c16

  Copy full SHA for 1661c16

Commits on Jan 8, 2019

• review response: use np.allclose instead of array_equals

  mpenkov

  committed
  72b1d81

  Copy full SHA for 72b1d81

• refactoring: simplify ngrams_weights matrix init

  mpenkov

  committed
  e467060

  Copy full SHA for e467060

• fixup: remove unused vectors_lockf attribute

  mpenkov

  committed
  c2740cd

  Copy full SHA for c2740cd

• fixup in _load_fasttext_model function

  mpenkov

  committed
  daa425a

  Copy full SHA for daa425a

• minor refactoring in unit tests

  mpenkov

  committed
  64844f3

  Copy full SHA for 64844f3

• adjust unit test

  Show description for 39e85f1
  mpenkov

  committed
  39e85f1

  Copy full SHA for 39e85f1

• temporarily disabling some assertions in tests

  mpenkov

  committed
  60d0477

  Copy full SHA for 60d0477

• document vectors_vocab_lockf and vectors_ngrams_lockf

  mpenkov

  committed
  52e2fbe

  Copy full SHA for 52e2fbe

• refactoring: further simplify growth of _lockf matrices

  mpenkov

  committed
  d08500b

  Copy full SHA for d08500b

• remove outdated comments

  mpenkov

  committed
  3a2f93e

  Copy full SHA for 3a2f93e

• fix deprecation warnings

  mpenkov

  committed
  3159a18

  Copy full SHA for 3159a18

Commits on Jan 9, 2019

• improve documentation for FastTextKeyedVectors

  mpenkov

  committed
  f262815

  Copy full SHA for f262815

• refactoring: extract L2 norm functions

  mpenkov

  committed
  b80c329

  Copy full SHA for b80c329

• add LoadFastTextFormatTest

  mpenkov

  committed
  127a13e

  Copy full SHA for 127a13e

• refactoring: remove old FB I/O code

  mpenkov

  committed
  2b96550

  Copy full SHA for 2b96550

• refactoring: FastTextKeyedVectors.init_post_load method

  mpenkov

  committed
  25ad1ae

  Copy full SHA for 25ad1ae

• refactoring: simplify init_post_load method

  mpenkov

  committed
  09388ec

  Copy full SHA for 09388ec

• refactoring: simplify init_post_load method

  mpenkov

  committed
  6054aa8

  Copy full SHA for 6054aa8

• refactoring: simplify calculate_vectors, rename to adjust_vectors

  mpenkov

  committed
  b92f435

  Copy full SHA for b92f435

• refactoring: simplify _lockf init

  mpenkov

  committed
  422e3b1

  Copy full SHA for 422e3b1

• remove old tests

  mpenkov

  committed
  553c8e0

  Copy full SHA for 553c8e0

• tox -e flake8

  mpenkov

  committed
  d42e506

  Copy full SHA for d42e506

• fixup: introduce OrderedDict to _fasttext_bin.py

  Show description for 425e942
  mpenkov

  committed
  425e942

  Copy full SHA for 425e942

• add unicode prefixes to literals for Py2.7 compatibility

  mpenkov

  committed
  802587a

  Copy full SHA for 802587a

• more Py2.7 compatibility stuff

  mpenkov

  committed
  ff82b71

  Copy full SHA for ff82b71

• refactoring: extract _process_fasttext_vocab function

  mpenkov

  committed
  dab47f3

  Copy full SHA for dab47f3

• still more Py2.7 compatibility stuff

  mpenkov

  committed
  914aa95

  Copy full SHA for 914aa95

• adding additional assertion

  mpenkov

  committed
  65abda9

  Copy full SHA for 65abda9

• re-enable disabled assertions

  mpenkov

  committed
  01d84d1

  Copy full SHA for 01d84d1

• delete out of date comment

  mpenkov

  committed
  611cdb2

  Copy full SHA for 611cdb2

• Revert "re-enable disabled assertions"

  Show description for 0c959a9
  mpenkov

  committed
  0c959a9

  Copy full SHA for 0c959a9

• more work on init_post_load function, update unit tests

  mpenkov

  committed
  c196ace

  Copy full SHA for c196ace

• update unit tests

  mpenkov

  committed
  fb51a6a

  Copy full SHA for fb51a6a

Commits on Jan 10, 2019

• review response: remove FastTextVocab class, keep alias

  mpenkov

  committed
  768a941

  Copy full SHA for 768a941

• review response: simplify _l2_norm_inplace function

  mpenkov

  committed
  f4643bb

  Copy full SHA for f4643bb

• review response: add docstring

  mpenkov

  committed
  d802e91

  Copy full SHA for d802e91

• review response: update docstring

  mpenkov

  committed
  92da774

  Copy full SHA for 92da774

• review response: move import

  mpenkov

  committed
  e638628

  Copy full SHA for e638628

• review response: adjust skip message

  mpenkov

  committed
  6e47a88

  Copy full SHA for 6e47a88

• Merge remote-tracking branch 'upstream/develop' into attrs

  mpenkov

  committed
  2cdad39

  Copy full SHA for 2cdad39

• reivew response: add test_hash_native

  mpenkov

  committed
  fbaf086

  Copy full SHA for fbaf086

• review response: explain how model was generated

  mpenkov

  committed
  dc32126

  Copy full SHA for dc32126

• review response: explain how expected values were generated

  mpenkov

  committed
  39e8844

  Copy full SHA for 39e8844

• review response: add test for long OOV word

  mpenkov

  committed
  08ee7d8

  Copy full SHA for 08ee7d8

• review response: remove unused comments

  mpenkov

  committed
  6d8a648

  Copy full SHA for 6d8a648

• review response: remove comment

  mpenkov

  committed
  734a0ac

  Copy full SHA for 734a0ac

• add test_continuation_load_gensim

  mpenkov

  committed
  143445e

  Copy full SHA for 143445e

• update model using gensim 3.6.0

  mpenkov

  committed
  250d388

  Copy full SHA for 250d388

• review response: get rid of struct_unpack

  Show description for 9fcf35e
  mpenkov

  committed
  9fcf35e

  Copy full SHA for 9fcf35e

• review response: implement handling for zero bucket edge case

  mpenkov

  committed
  c1aeb85

  Copy full SHA for c1aeb85

• review response: add test_save_load

  mpenkov

  committed
  58c1166

  Copy full SHA for 58c1166

• review response: add test_save_load_native

  mpenkov

  committed
  e5960ed

  Copy full SHA for e5960ed

Commits on Jan 11, 2019

• workaround appveyor tempfile issue

  mpenkov

  committed
  52230aa

  Copy full SHA for 52230aa

• fix tests

  menshikh-iv

  committed
  14c497d

  Copy full SHA for 14c497d