[FastTokenizer] Add unittest for fast_tokenizer

[joey12300]

PR types

Bug fixes

PR changes

Others

Description

Add unittest for fast_tokenizer

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

Merging #4339 (bbc999a) into develop (bd2be0f) will increase coverage by 2.38%.
The diff coverage is 96.15%.

@@             Coverage Diff             @@
##           develop    #4339      +/-   ##
===========================================
+ Coverage    36.30%   38.69%   +2.38%     
===========================================
  Files          419      422       +3     
  Lines        59254    59661     +407     
===========================================
+ Hits         21514    23087    +1573     
+ Misses       37740    36574    -1166     

Impacted Files	Coverage Δ
paddlenlp/transformers/tokenizer_utils_fast.py	77.77% <50.00%> (ø)
paddlenlp/transformers/auto/tokenizer.py	81.74% <100.00%> (+13.49%)	⬆️
paddlenlp/transformers/bert/fast_tokenizer.py	100.00% <100.00%> (+100.00%)	⬆️
paddlenlp/transformers/convert_slow_tokenizer.py	89.43% <100.00%> (+89.43%)	⬆️
paddlenlp/transformers/ernie/fast_tokenizer.py	100.00% <100.00%> (+100.00%)	⬆️
paddlenlp/transformers/ernie_m/fast_tokenizer.py	90.90% <100.00%> (+90.90%)	⬆️
paddlenlp/transformers/tinybert/fast_tokenizer.py	100.00% <100.00%> (+100.00%)	⬆️
paddlenlp/transformers/unimo/modeling.py	81.66% <0.00%> (-0.92%)	⬇️
paddlenlp/transformers/roberta/modeling.py	89.85% <0.00%> (-0.37%)	⬇️
...s/fast_transformer/transformer/fast_transformer.py	12.08% <0.00%> (-0.34%)	⬇️
... and 35 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[zjjlivein]

单测CI 通过

[sijunhe]

高质量的PR，点赞！几个小问题

[sijunhe]

所有的单测我建议把tokenizer和fast_tokenizer分开，即分test_tokenizer和test_fast_tokenizer两个method. 如果有错，比较容易debug

[joey12300]

Done，新增test_fast_and_python_full_tokenizer单测，用于比较fast和python的tokenizer的输出是否一致。

[sijunhe]

这个PR我ok的，@wj-Mcat 也看看

[wj-Mcat]

其它我觉得都 OK，就包名的小问题。

另外，fast-tokenizer的自动化打包发包的事情后续是否有必要添加到github action workflow中去吗？github action 中支持os matrix 来提供多平台自动化打包编译，不过最后还是看你们的安排。

[wj-Mcat]

我查了下，包名应该是：fast-tokenizer-python，具体可见：https://pypi.org/project/fast-tokenizer-python/

我本地运行: pip install fast_tokenizer_python 会直接报错。