You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Earlier today our ElasticSearch servers hit an OutOfMemory issue. By default, Elasticsearch is configured to do a heap dump whenever it hits an out of memory issue. When the out of memory error happened earlier today, the default disk partition used for writing the heap dump happened to be smaller than the heap dump file, which resulted in that disk partition completely filling up. Other processes on the server weren't affected by this partition filling up, but this still isn't great to have a fill disk partition.
Since we're probably not too interested in inspecting the heap dumps in these out of memory situations, I've disabled the heap dumps from occurring in the future, which should prevent the disk partitions from filling up: NREL/api-umbrella@920621e
The real underlying issue is why Elasticsearch hit an hard OutOfMemory error to begin with, but it looks like there was an odd culmination of separate admins from different agencies all performing analytics queries over long durations in the admin at the same time. If this happens again, we can try doing some more elasticsearch tuning, but the better answer is probably the fabled analytics task (#235... sigh.. which, unfortunately I haven't had time to wrap up, but I'll try to revisit that soon to see if we can finally, finally, finally push that over the finish line).
The text was updated successfully, but these errors were encountered:
Earlier today our ElasticSearch servers hit an OutOfMemory issue. By default, Elasticsearch is configured to do a heap dump whenever it hits an out of memory issue. When the out of memory error happened earlier today, the default disk partition used for writing the heap dump happened to be smaller than the heap dump file, which resulted in that disk partition completely filling up. Other processes on the server weren't affected by this partition filling up, but this still isn't great to have a fill disk partition.
Since we're probably not too interested in inspecting the heap dumps in these out of memory situations, I've disabled the heap dumps from occurring in the future, which should prevent the disk partitions from filling up: NREL/api-umbrella@920621e
The real underlying issue is why Elasticsearch hit an hard OutOfMemory error to begin with, but it looks like there was an odd culmination of separate admins from different agencies all performing analytics queries over long durations in the admin at the same time. If this happens again, we can try doing some more elasticsearch tuning, but the better answer is probably the fabled analytics task (#235... sigh.. which, unfortunately I haven't had time to wrap up, but I'll try to revisit that soon to see if we can finally, finally, finally push that over the finish line).
The text was updated successfully, but these errors were encountered: