Enable dropout = 0.0 as an equivalent to none in BPE

[mcognetta]

This is related to the discussion in #1541.

This PR allows for 0.0 to be used as the dropout value in BPE models with equivalent functionality to none. Previously, the docs and implementation were inconsistent:

• the docs said 0.0 to 1.0 were acceptable values for dropout and that the default was 0
  • the default was actually none
• it was possible to set dropout to 0.0 once the model was built (and it would work the same as if it was none)
  • however, in this case caching was disabled
  • on a small benchmark, i measured a 50% increase in encoding time with dropout = 0.0 compared to none (this discrepancy goes away after this PR is applied)
• serialization worked, but deserialization failed, since there was a check that dropout \in (0.0, 1.0]
• initializing a python BPE model with dropout 0.0 failed (e.g. BPE(dropout = 0.0))

This simply allows for 0.0 to be an acceptable value during initialization and enables caching when tokenizing if dropout == 0.0.

E.g., now the following works

>>> from tokenizers import Tokenizer, models
>>> tokenizer = Tokenizer(models.BPE(dropout = 0.0))
>>> s = tokenizer.to_str()
>>> s
'{"version":"1.0","truncation":null,"padding":null,"added_tokens":[],"normalizer":null,"pre_tokenizer":null,"post_processor":null,"decoder":null,"model":{"type":"BPE","dropout":0.0,"unk_token":null,"continuing_subword_prefix":null,"end_of_word_suffix":null,"fuse_unk":false,"byte_fallback":false,"ignore_merges":false,"vocab":{},"merges":[]}}'
>>> deserialized = Tokenizer.from_str(s)
>>> deserialized.model.dropout
0.0

whereas before it errored.

---

As future work, I think that dropout should be made non-optional, with the default being 0.0. This would remove the checks for dropout.is_none(), etc, but keep the functionality the same. However, I guess this would be a breaking change (since then all tokenizers serialized before this change would be invalid?).

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

LGTM! Thanks for adding tests in python and rust.

[ArthurZucker]

rebasing on main should fix clippy issues

[mcognetta]

Thanks. I fixed one of the lint errors which was a range readability thing.

[mcognetta]

Fixed one more formatting issue. Now I think it should be all good!

[ArthurZucker]

thanks! Le't s remove the unrelated change (maybe from rebasing ? )

[ArthurZucker]

Actually this should not be in the diff

[mcognetta]

I goofed, and now its getting worse 😬

[mcognetta]

Ok, it's all good now, unless you want me to squash the commits.

[ArthurZucker]

No don't worry, this LGMT! thanks for updating