How can I explicitly profile the memory used for each part of the model? #3080

dlin511 started this conversation in General

dlin511
Jan 23, 2025

Hi,

Does lmdeploy/turbomind engine support any way to profile the memory usage of each part of the model?

For example, can I see how much memory the model is occupying vs the activation states vs the KV cache?

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment