Concurrency safety issues in Triton Server streaming inference

[liuyang-my]

Is there a thread safety issue in ModelStreamInferHandler of Triton Server grpc in streaming mode? After the Process function handles the WRITEREADY state, the state transitions to WRITTEN. If HasReadyResponse is evaluated as false during the handling of WRITTEN, the state is set to ISSUED. If StreamInferResponseComplete is called before the state is modified to ISSUED, and StreamInferResponseComplete checks that is_decoupled_ is true and step_ is WRITTEN, it does nothing. In this case, when the thread context switches back to Process, the state is set to ISSUED. If this response happens to be the last response and there are no tasks in the CQ, will Finish never be executed?

[dyastremsky]

Thanks for identifying and reporting this. If this is a race condition, we have not yet seen it reported. It's good to be proactive though.

CC: @tanmayv25, who wrote this code. It's 3 years old, so a lot of context might be gone. Let me know if you want me to convert this into an issue and create a ticket for us to investigate.

[liuyang-my]

Thank you for your response. Our company is currently using Triton Server at a large scale. We also have some LLM scenarios utilizing the streaming functionality. So far, we haven't encountered any issues. However, for the sake of production environment stability, while reviewing the code, we noticed this particular concern and would like to understand if there is any potential risk of thread safety in this area.

[dyastremsky]

Of course. We greatly appreciate you reviewing the code and relaying this to us. Converting to issue for tracking purposes: #6245.

We can continue discussion there.