[Frontend] Refactor prompt processing

[DarkLight1337]

This PR refactors various parts of the OpenAI-compatible server:

• The _validate_prompt_and_tokenize method has been decomposed so that prompt and prompt_ids are processed separately.
• The logging of prompt and prompt_ids has been moved from vllm.AsyncLLMEngine to vllm.entrypoints.logger.RequestLogger such that redundant data is no longer passed into the core engine. This also enables logging for tokenization endpoints.
• Explicitly handle LoRA and prompt adapters for embedding endpoints. Previously, they were silently ignored.
• Use a different prefix for request_id based on the endpoint type:
  • Completion API: cmpl-* (as before)
  • Chat Completion API: chat-*
  • Embeddings API: embd-*
  • Tokenization API: tokn-*
• Fixed various type errors

[DarkLight1337]

@njhill I see that you are working on #3512. I suppose it would be better to get that merged before continuing with this refactor.

[DarkLight1337]

Seems that #4032 fixed the LoRA bugs, however entrypoints-test is still failing.

[DarkLight1337]

Update: I found that it is due to a bug in my refactored parsing, my bad. I have fixed it just now.

[DarkLight1337]

@njhill The tests now pass. To move on with #3978, perhaps we can merge this PR first?

[DarkLight1337]

Update: I found that it is due to a bug in my refactored parsing, my bad. I have fixed it just now.

I'm updating test_batch_completions to better catch this error. Previously, the error only occurred in test_echo_logprob_completion which misled me from the real cause.

[njhill]

Looks like this line also needs to be removed from the tokenization test.

[DarkLight1337]

Looks like this line also needs to be removed from the tokenization test.

Actually, let me add add_special_tokens to Completions API (defaulting to True to maintain the existing behaviour) as well to make things consistent. It's a vLLM feature originally introduced by #5278.

[DarkLight1337]

My only concern is whether some folks might make use of the logging in the engine with different front ends.

I've moved out the logging to a separate class vllm.entrypoints.logger.RequestLogger, so it should be easy for them to add code to log the requests if need be.

[DarkLight1337]

I have finished addressing your comments.

[njhill]

Thanks @DarkLight1337!