Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent Elasticsearch OutOfMemory issues from filling up disks #351

Closed
GUI opened this issue Aug 9, 2016 · 0 comments
Closed

Prevent Elasticsearch OutOfMemory issues from filling up disks #351

GUI opened this issue Aug 9, 2016 · 0 comments

Comments

@GUI
Copy link
Member

GUI commented Aug 9, 2016

Earlier today our ElasticSearch servers hit an OutOfMemory issue. By default, Elasticsearch is configured to do a heap dump whenever it hits an out of memory issue. When the out of memory error happened earlier today, the default disk partition used for writing the heap dump happened to be smaller than the heap dump file, which resulted in that disk partition completely filling up. Other processes on the server weren't affected by this partition filling up, but this still isn't great to have a fill disk partition.

Since we're probably not too interested in inspecting the heap dumps in these out of memory situations, I've disabled the heap dumps from occurring in the future, which should prevent the disk partitions from filling up: NREL/api-umbrella@920621e

The real underlying issue is why Elasticsearch hit an hard OutOfMemory error to begin with, but it looks like there was an odd culmination of separate admins from different agencies all performing analytics queries over long durations in the admin at the same time. If this happens again, we can try doing some more elasticsearch tuning, but the better answer is probably the fabled analytics task (#235... sigh.. which, unfortunately I haven't had time to wrap up, but I'll try to revisit that soon to see if we can finally, finally, finally push that over the finish line).

@GUI GUI closed this as completed Aug 9, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant