Change our tokenizer a bit to be more accurate

[dkotter]

Description of the Change

We have a lightweight tokenizer class that attempts to determine how many tokens are in a string. This is done mostly by determining how many characters are in the string and how many characters are in a single token. This is not meant to be 100% accurate but is meant to be close enough for our use case (which is ensuring we stay within the token limits of our model).

Recently I ran across an article that was super long and I received a token length error when trying to process it. In debugging, I found we weren't being aggressive enough with the counting of tokens, and thus we weren't trimming enough of the content to stay within the model limits.

This PR lowers the number of characters per token from 4 to 3.5. This fixed the issue I ran into and seems to be more accurate in counting tokens.

How to test the Change

Send content to OpenAI (either generate an excerpt or generate titles) and ensure things still work and no errors are shown

Changelog Entry

Fixed - More accurate token counts when trimming content

Credits

Props @dkotter

Checklist:

• I agree to follow this project's Code of Conduct.
• I have updated the documentation accordingly.
• I have added tests to cover my change.
• All new and existing tests pass.

[faisal-alvi]

Thanks for the PR and brief details. Tested the plugin features and found are working as expected in the fix branch, no errors are shown.

Title Generation

Excerpt Generation

Post Classification