Code

Add option to use fast HF tokenizer.#482

Code

Merged

tholor merged 32 commits intodeepset-ai:masterdeepset-ai/FARM:masterfrom PhilipMay:add_fast_tokenizerPhilipMay/FARM:add_fast_tokenizerCopy head branch name to clipboardSep 2, 2020

+420-93

Commits

Commits on Aug 1, 2020

• Add option to use fast HF tokenizer

  PhilipMay

  committed
  651463a

  Copy full SHA for 651463a

• Hand merge tests from PR deepset-ai#205

  PhilipMay

  committed
  a433483

  Copy full SHA for a433483

• test_inferencer_with_fast_bert_tokenizer

  PhilipMay

  committed
  c20c5db

  Copy full SHA for c20c5db

• test_fast_bert_tokenizer

  PhilipMay

  committed
  5f2b5ee

  Copy full SHA for 5f2b5ee

• test_fast_bert_tokenizer_strip_accents

  PhilipMay

  committed
  fa3bd67

  Copy full SHA for fa3bd67

• test_fast_electra_tokenizer

  PhilipMay

  committed
  cd7298c

  Copy full SHA for cd7298c

• Fix OOM issue of CI

  Show description for 01e5ffb
  PhilipMay

  committed
  01e5ffb

  Copy full SHA for 01e5ffb

Commits on Aug 2, 2020

• Extend test for fast tokenizer

  Show description for 42f345f
  PhilipMay

  committed
  42f345f

  Copy full SHA for 42f345f

• test_fast_tokenizer for more model typed

  Show description for 9b021ff
  PhilipMay

  committed
  9b021ff

  Copy full SHA for 9b021ff

• Fix tokenize_with_metadata

  PhilipMay

  committed
  86d7fd5

  Copy full SHA for 86d7fd5

• Split tokenizer tests

  PhilipMay

  committed
  a8f4638

  Copy full SHA for a8f4638

• Fix pytest params bug in test_tok

  PhilipMay

  committed
  cdccafa

  Copy full SHA for cdccafa

Commits on Aug 4, 2020

• Fix fast tokenizer usage

  PhilipMay

  committed
  47d4b6a

  Copy full SHA for 47d4b6a

• add missing newline eof

  PhilipMay

  committed
  8318063

  Copy full SHA for 8318063

• Add test fast tok. doc_callif.

  PhilipMay

  committed
  8c61e3b

  Copy full SHA for 8c61e3b

• Remove RobertaTokenizerFast

  PhilipMay

  committed
  aec7d2d

  Copy full SHA for aec7d2d

• Fix Tokenizer load and save.

  PhilipMay

  committed
  75ea9dd

  Copy full SHA for 75ea9dd

• Fix typo

  PhilipMay

  committed
  2d2cd00

  Copy full SHA for 2d2cd00

Commits on Aug 5, 2020

• Improve test test_embeddings_extraction

  Show description for 8afa136
  PhilipMay

  committed
  8afa136

  Copy full SHA for 8afa136

• Dosctring for fast tokenizers improved

  PhilipMay

  committed
  042fde0

  Copy full SHA for 042fde0

• tokenizer_args docstring

  PhilipMay

  committed
  7ed385f

  Copy full SHA for 7ed385f

• Extend test_embeddings_extraction to fast tok.

  PhilipMay

  committed
  d4eb59c

  Copy full SHA for d4eb59c

• extend test_ner with fast tok.

  PhilipMay

  committed
  4f87604

  Copy full SHA for 4f87604

Commits on Aug 6, 2020

• fix sample_to_features_ner for fast tokenizer

  tholor

  committed
  bc7abca

  Copy full SHA for bc7abca

• temp fix for is_pretokenized until fixed upstream

  tholor

  committed
  da9c2f5

  Copy full SHA for da9c2f5

Commits on Aug 25, 2020

• Make use of fast tokenizer possible + fix bug in offset calculation

  bogdankostic

  committed
  19cc211

  Copy full SHA for 19cc211

• Merge branch 'master' into add_fast_tokenizer

  bogdankostic

  authored
  6d0a3c1

  Copy full SHA for 6d0a3c1

Commits on Aug 31, 2020

• Make fast tokenization possible with NER, LM and QA

  bogdankostic

  committed
  7e75de1

  Copy full SHA for 7e75de1

• Merge remote-tracking branch 'origin/add_fast_tokenizer' into add_fast_tokenizer

  bogdankostic

  committed
  0e4b1b0

  Copy full SHA for 0e4b1b0

Commits on Sep 1, 2020

• Change error messages

  bogdankostic

  committed
  eb46629

  Copy full SHA for eb46629

• Add tests

  bogdankostic

  committed
  06d51c0

  Copy full SHA for 06d51c0

Commits on Sep 2, 2020

• update error messages, comments and truncation arg in tokenizer

  tholor

  committed
  1acaff4

  Copy full SHA for 1acaff4