Fix tokenizer write on read only file system

[EricLBuehler]

No description provided.

[github-actions]

Code Metrics Report

  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Dockerfile              1           34           25            0            9
 Happy                   1          442          369            0           73
 JSON                    9           21           21            0            0
 Python                 24          864          731           25          108
 TOML                   15          402          364            1           37
-------------------------------------------------------------------------------
 Jupyter Notebooks       1            0            0            0            0
 |- Markdown             1           60           30           22            8
 |- Python               1           96           87            1            8
 (Total)                            156          117           23           16
-------------------------------------------------------------------------------
 Markdown               16         1056            0          782          274
 |- BASH                 6          203          190            0           13
 |- Python               6          121          110            0           11
 |- Rust                 3          185          172            9            4
 (Total)                           1565          472          791          302
-------------------------------------------------------------------------------
 Rust                   91        29568        26948          424         2196
 |- Markdown            46          467            0          454           13
 (Total)                          30035        26948          878         2209
===============================================================================
 Total                 159        32387        28458         1232         2697
===============================================================================
  

[polarathene]

Not sure if this is relevant to the original reason for the utility, but a recent change in tokenizers ignores added tokens in the decoder. Which may affect the relevance of this utility / workaround? 🤷‍♂️

---

For context on this utility method, while the logic itself is pretty clear that it adds the token content field as a lookup key for the associated ID field of the token object in added_tokens, it may be helpful to have a little reference of the file content. So if anyone lands here via git blame I documented the following here (collapsed section at end of linked comment):

Relevant context snippets (click to view)

The utility itself is to avoid these log lines (related discussion), that I only seemed to trigger with a llama3 tokenizer.json:

Similar to what is found in added_tokens.json (this file is at least present in NousResearch/Hermes-2-Pro-Mistral-7B which lacks a tokenizer.json and has tokenizer.model instead).

[ArthurZucker]

The added_tokens.json is an old format btw

[EricLBuehler]

The added_tokens.json is an old format btw

We do not use added_tokens.json as a backend, but thank you for letting us know!