What's Changed
⚡ Now tokenicer instance dynamically inherits the native tokenizer.__class__ of tokenizer passed in or loaded via our Tokenicer.load() api.
⚡ CI now tests tokenizers from 64 models
- fix mpt pad token bug by @CL-ModelCloud in #24
- fix model_config bugs by @CL-ModelCloud in #25
- test code clean up by @CL-ModelCloud in #26
- Inherits PretrainedTokenizer by @Qubitium in #28
- loop & test all models by @CSY-ModelCloud in #30
Full Changelog: v0.0.2...v0.0.4
Contributors
Qubitium, CL-ModelCloud, and CSY-ModelCloud