You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As far as I am aware, I have moved all my Pods to using tmpfs for file access, but nevertheless, when I launch 90 Pods (all with the same Docker image) at once, the system locks up because of excessive disk I/O on the EBS volume. How do I debug this?
On AL2 I can use tools like dstat, atop, and iotop to look at what is going on, but these tools are not installed or easily installed (dstat and iotop are both Python programs) on Bottlerocket, and why I tried running them in a privileged container, I still didn't get host stats, as far as I could tell. Then again, this is at the outer limits of my training and experience, so while you don't need to ELi5, I would appreciate some clear directions and/or advice as to how to figure out what is sucking up all the disk I/O and how to move it to tmpfs.
In a related issue, when running EKS on Bottlerocket, how do I get it to reserve memory I allocate to tmpfs for this purpose, so that Kubernetes does not try to allocate it to Pods?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
This is a follow-up to #4072
As far as I am aware, I have moved all my Pods to using
tmpfs
for file access, but nevertheless, when I launch 90 Pods (all with the same Docker image) at once, the system locks up because of excessive disk I/O on the EBS volume. How do I debug this?On AL2 I can use tools like
dstat
,atop
, andiotop
to look at what is going on, but these tools are not installed or easily installed (dstat
andiotop
are both Python programs) on Bottlerocket, and why I tried running them in a privileged container, I still didn't get host stats, as far as I could tell. Then again, this is at the outer limits of my training and experience, so while you don't need to ELi5, I would appreciate some clear directions and/or advice as to how to figure out what is sucking up all the disk I/O and how to move it totmpfs
.In a related issue, when running EKS on Bottlerocket, how do I get it to reserve memory I allocate to
tmpfs
for this purpose, so that Kubernetes does not try to allocate it to Pods?Beta Was this translation helpful? Give feedback.
All reactions