Prediction request timeout including time spent in queue

[nateagr]

Is your feature request related to a problem? Please describe.

The feature request is not related to a bug. But at my company, we have this following problem: we would like to be able to dismiss prediction requests after a certain amount of time. Why ? Because they are not relevant anymore after this amount of time. As of today, it is possible to define a timeout, responseTimeout (https://pytorch.org/serve/configuration.html), but it only applies to model inference (handler). It does not include the time spent in queue.

Describe the solution

To avoid any regression, I would suggest to add a new model parameter: predictionRequestTimeout. This parameter defines the timeout after which a prediction request is dismissed. It includes the time spend in queue and inference time. From an implementation point of view, I would suggest to modify:

• pollBatch method of Model (https://github.com/pytorch/serve/blob/master/frontend/server/src/main/java/org/pytorch/serve/wlm/Model.java#L174) to dismiss prediction requests that are still in queue and too old wrt to predictionRequestTimeout.
• run method of WorkerThread (https://github.com/pytorch/serve/blob/master/frontend/server/src/main/java/org/pytorch/serve/wlm/WorkerThread.java#L189) to timeout after min(responseTimeout, remaining time of predictionRequestTimeout) instead of after responseTimeout.

If you agree with this feature requests, I'm willing to push a PR.

[nateagr]

Hello there! Should I suggest a PR ? Or do you need time to think about it ?

[HamidShojanazeri]

@nateagr please go a head with a PR, we would appreciate it and happy to review and help to merge.