Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tflite interpreter for runtime inference #48

Closed
cpmpercussion opened this issue Aug 23, 2024 · 0 comments · Fixed by #50
Closed

tflite interpreter for runtime inference #48

cpmpercussion opened this issue Aug 23, 2024 · 0 comments · Fixed by #50
Labels
enhancement New feature or request

Comments

@cpmpercussion
Copy link
Owner

With the capacity to save models as tflite files we have experimented with making predictions using Tensorflow Lite.

On a powerful laptop, the present Keras model inference has an average time of 4.96ms (median 4.98) over 3200 predictions. Over 10000 predictions with a 9d s network, the tflite inference runner averages 0.17ms per prediction.

29.17x improvement!

Far too much performance improvement to leave on the table. This issue proposes building out the tflite inference engine so that it is automatically is used if a user specifies a .tflite file in config.toml.

If tflite inference seems to behave similarly, we should be doing this by default.

Thanks Scott H for helping to surface this.

@cpmpercussion cpmpercussion added the enhancement New feature or request label Aug 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant