Fix kv_cache_type issue

[qingquansong]

Fix kv_cache_type issue related to #1930

Details described: #1930 (comment)

[qingquansong]

@Barry-Delaney ^^ mind helping take a look? Seems to help resolve some issue here #1930 Thank you!

[lfr-0531]

Thanks for the fix. We'll merge you changes into internal code base.

[hchings]

Closing this out as it's been merged.

[kaiyux]

Hi @qingquansong , thanks a lot for the contribution! Your changes will be included in the next main branch update, and we'll mark you as co-author.

Please also note that, the Python benchmark is not suggested to be used and will soon be deprecated. Please take a look at the on-going support to a benchmarking suite, as well as the C++ benchmark for the support to the latest features.

Closing this out as it's been merged.

@hchings To clarify in case there are going to be confusion - the changes are merged in the internal repo, but not updated to the external GitHub repo yet. For future cases I would suggest to only close the PR after we pushed the main branch update that includes the changes. Please let me know if there are any questions, thanks!

Thanks again for your support!

[qingquansong]

Hi @qingquansong , thanks a lot for the contribution! Your changes will be included in the next main branch update, and we'll mark you as co-author.

Please also note that, the Python benchmark is not suggested to be used and will soon be deprecated. Please take a look at the on-going support to a benchmarking suite, as well as the C++ benchmark for the support to the latest features.

Closing this out as it's been merged.

@hchings To clarify in case there are going to be confusion - the changes are merged in the internal repo, but not updated to the external GitHub repo yet. For future cases I would suggest to only close the PR after we pushed the main branch update that includes the changes. Please let me know if there are any questions, thanks!

Thanks again for your support!

Sound great. Thank you! Besides the C++ throughput API benchmarking, I'm currently also switching to using the hlapi and benchmarking it with this perf evaluator script, is the a suggested one to use? It contains both latency and throughput results which is quite nice to use and the only thing that I'm modifying to add is the concurrency part (which I'm planning to use the Poisson request). Not sure if you think that's a good feature to add here .

[kaiyux]

Hi @qingquansong , thanks a lot for the contribution! Your changes will be included in the next main branch update, and we'll mark you as co-author.
Please also note that, the Python benchmark is not suggested to be used and will soon be deprecated. Please take a look at the on-going support to a benchmarking suite, as well as the C++ benchmark for the support to the latest features.

Closing this out as it's been merged.

@hchings To clarify in case there are going to be confusion - the changes are merged in the internal repo, but not updated to the external GitHub repo yet. For future cases I would suggest to only close the PR after we pushed the main branch update that includes the changes. Please let me know if there are any questions, thanks!
Thanks again for your support!

Sound great. Thank you! Besides the C++ throughput API benchmarking, I'm currently also switching to using the hlapi and benchmarking it with this perf evaluator script, is the a suggested one to use? It contains both latency and throughput results which is quite nice to use and the only thing that I'm modifying to add is the concurrency part (which I'm planning to use the Poisson request). Not sure if you think that's a good feature to add here .

@qingquansong Thanks a lot for your attention on those details! The tests/hlapi/hlapi_evaluator.py is under tests directory, and currently used for test purpose only. I would still suggest to try the suite and feel free to let us know your feedback. We'll consolidate the benchmarking workflows.

[DanBlanaru]

Hello @qingquansong. I was in charge of the publishing for this week.
It seems GitHub did not like the formatting as it did not properly credit you.
I apologize, I will credit you in next week's push.

[DanBlanaru]

Hello @qingquansong, we credited you in the last push to main .Thank you for the contribution again!