Allow the injection of TokenCountEstimator

[andreadimaio]

This PR resolves #697.

The idea is to allow the injection of the TokenCountEstimator interface, which can be used in different ways, for example to know in advance the cost of processing a certain text, or to choose one model or another based on the maximum number of tokens.

Today, when I look at the code to see which LLM providers implement this interface, I see

• bam
• watsonx
• azure-openai

For the azure-openai I found something strange, the tokenizer variable is never set, so when creating tests I always get a NullPointerException(There are some TODOs in the code to understand what the points are).

As far as I understand, the correct way to create the tokenizer for azure-openai is to use something like:

new AzureOpenAiTokenizer(<modelId>)

but I'm not able to test this, so what I've done is just put some TODOs in the code.

Is there anyone who can look into this? Does it make sense to have a TokenCountEstimator for azure-openai or should I just ignore it for now?

[geoand]

Is there anyone who can look into this? Does it make sense to have a TokenCountEstimator for azure-openai or should I just ignore it for now?

Maybe @csotiriou knows :)

[geoand]

Very nice!