Experimental: tokenizers with and without templates #168
Conversation
| try applyChatTemplate(messages: messages, chatTemplate: .literal(chatTemplate), addGenerationPrompt: true, truncation: false, maxLength: nil, tools: nil) | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
See comment in TokenizersTemplates
| ] | ||
|
|
||
| open class PreTrainedTokenizerWithTemplates : PreTrainedTokenizer { | ||
| // I don't know why these need to be here. They are implemented in the protocol, **and** in the superclass. |
There was a problem hiding this comment.
Yes, if these overrides don't exist, the linker can't find the implementations.
| import Foundation | ||
| import Hub | ||
|
|
||
| @_exported import TokenizersCore |
There was a problem hiding this comment.
This is a good chunk of the magic. The new Tokenizers implementation is just this wrapper file, which exposes the imported TokenizersCore types.
| #if canImport(TokenizersTemplates) | ||
| import TokenizersTemplates | ||
| public typealias PreTrainedTokenizer = PreTrainedTokenizerWithTemplates | ||
| #endif |
There was a problem hiding this comment.
So if TokenizersTemplates is available (because users have declared it as a dependency), we override the definition so the factory below uses the subclass.
| } | ||
|
|
||
| // See https://github.com/xenova/transformers.js/blob/1a9964fb09b8f54fcbeac46dc6aae8d76795809d/src/tokenizers.js#L3203 for these exceptions | ||
| class LlamaPreTrainedTokenizer: PreTrainedTokenizer { |
There was a problem hiding this comment.
This could be moved back to TokenizersCore, but then we'd need a typealias here as well (and a subclass).
|
cc @greenrazer @FL33TW00D @Vaibhavs10 for opinions and feedback. |
This builds on top of #166 by @greenrazer.
The goal is to be able to opt-in to the chat template feature, which carries the Jinja dependency, which in turn depends on swift-collections and whatnot. It works in its current state, but it's ugly – I found way more cross-module quirks than I was expecting. I wanted to make it as easy as possible for consumers and allow to use either version (with or without templates) from their SPM manifests.
Opinions, corrections, and alternative ideas are encouraged and most welcome!
How to use:
Add the following dependency to your target, as usual (except
Tokenizersis now a product, no need to use the fullTransformerslib). This applies to projects such as WhisperKit.This applies to projects such as mlx-swift-examples.
dependencies: [ .product(name: "Tokenizers", package: "swift-transformers"), + .product(name: "TokenizersTemplates", package: "swift-transformers"), ],That's it, you don't need to import
TokenizersTemplatesor do anything other than just declaring it as a dependency.Known issues: