quantized models

[marsupialtail]

Does this support quantized models by any chance?

[priyanksonis]

@marsupialtail yes, I tried , you can quantize the onnx models like this, then use quantized model for inference

from onnxruntime.quantization import quantize_dynamic, QuantType

#quantize encoder
model_input = "temp1/t5-own--encoder.onnx"
model_output = "temp1_compressed_onnxruntime/t5-own--encoder.onnx"

quantize_dynamic(model_input, model_output, weight_type=QuantType.QInt8)

#quantize decoder
model_input = "temp1/t5-own--decoder-with-lm-head.onnx"
model_output = "temp1_compressed_onnxruntime/t5-own--decoder-with-lm-head.onnx"

quantize_dynamic(model_input, model_output, weight_type=QuantType.QInt8)