Code

Add-legacy-tests#1597

Code

Merged

ArthurZucker merged 14 commits intomainhuggingface/tokenizers:mainfrom add-legacy-testshuggingface/tokenizers:add-legacy-testsCopy head branch name to clipboardAug 6, 2024

+77

Commits

Commits on Aug 6, 2024

• add tests

  ArthurZucker

  committed
  021eec2

• decoder as well

  ArthurZucker

  committed
  6860310

• check error

  ArthurZucker

  committed
  b580f96

• propagate

  ArthurZucker

  committed
  0711bc6

• lint

  ArthurZucker

  committed
  f306795

• rafiune the test

  ArthurZucker

  committed
  4ecd4d0

• lint

  ArthurZucker

  committed
  9a82614

• revert decoder changes

  ArthurZucker

  committed
  d14cc7d

• on more?

  ArthurZucker

  committed
  669a9e8

• fmt

  ArthurZucker

  committed
  8c642ee

• Update tokenizers/src/pre_tokenizers/mod.rs

  Show description for d5dc8ce
  

  ArthurZucker

  and

  Narsil

  authored
  d5dc8ce

• fix commit

  ArthurZucker

  committed
  002bb46

• simplify err

  ArthurZucker

  committed
  81b1ecb

• fmt

  ArthurZucker

  committed
  14dba86