You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With the capacity to save models as tflite files we have experimented with making predictions using Tensorflow Lite.
On a powerful laptop, the present Keras model inference has an average time of 4.96ms (median 4.98) over 3200 predictions. Over 10000 predictions with a 9d s network, the tflite inference runner averages 0.17ms per prediction.
29.17x improvement!
Far too much performance improvement to leave on the table. This issue proposes building out the tflite inference engine so that it is automatically is used if a user specifies a .tflite file in config.toml.
If tflite inference seems to behave similarly, we should be doing this by default.
Thanks Scott H for helping to surface this.
The text was updated successfully, but these errors were encountered:
With the capacity to save models as tflite files we have experimented with making predictions using Tensorflow Lite.
On a powerful laptop, the present Keras model inference has an average time of 4.96ms (median 4.98) over 3200 predictions. Over 10000 predictions with a 9d s network, the tflite inference runner averages 0.17ms per prediction.
29.17x improvement!
Far too much performance improvement to leave on the table. This issue proposes building out the tflite inference engine so that it is automatically is used if a user specifies a
.tflite
file inconfig.toml
.If tflite inference seems to behave similarly, we should be doing this by default.
Thanks Scott H for helping to surface this.
The text was updated successfully, but these errors were encountered: