You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Support loading models with multiple signatures and selection of a particular signature during model inference. This is supported by standard inference servers like TF Serving and libraries like libtensorflow & libtorch.
Our models have multiple methods (PyTorch) or signatures (TensorFlow) and based on the operation we perform, we have to choose the correct signature at inference time.
Describe the solution you'd like
Support loading multiple signatures at model loading time for Tensorflow, Torch and any backend as long as it supports it.
During inference time, fix a InferenceRequest.parameter key "SIGNATURE_NAME", when provided, will allow the backend to run the model on that signature. In PyTorch that would be "METHOD_NAME".
Describe alternatives you've considered
Instead of using Triton, using the model framework libraries directly such as libtensorflow and libtorch.
Additional context
Simple example is a model can do a forward pass, a forward pass with additional ranking and produce new feature tensor and these "signatures" can share most of the model graph so having them in one model is our desire.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Support loading models with multiple signatures and selection of a particular signature during model inference. This is supported by standard inference servers like TF Serving and libraries like libtensorflow & libtorch.
Our models have multiple methods (PyTorch) or signatures (TensorFlow) and based on the operation we perform, we have to choose the correct signature at inference time.
Describe the solution you'd like
Describe alternatives you've considered
Instead of using Triton, using the model framework libraries directly such as libtensorflow and libtorch.
Additional context
Simple example is a model can do a forward pass, a forward pass with additional ranking and produce new feature tensor and these "signatures" can share most of the model graph so having them in one model is our desire.
The text was updated successfully, but these errors were encountered: