You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently tokenizer and token representation classes does not support dumping to some easy to compare format which could be used for testing. Additionally there is not simple way to add native implementation of tokenizer to provide optional speedup of tokenization process. Currently the idea would be to extract token filling logic from token classes and make them as simple containers as possible, hence allowing construction from C/C++/Rust implementation of tokenizer.
The text was updated successfully, but these errors were encountered:
Currently tokenizer and token representation classes does not support dumping to some easy to compare format which could be used for testing. Additionally there is not simple way to add native implementation of tokenizer to provide optional speedup of tokenization process. Currently the idea would be to extract token filling logic from token classes and make them as simple containers as possible, hence allowing construction from C/C++/Rust implementation of tokenizer.
The text was updated successfully, but these errors were encountered: