-
-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OOM: mdedup
hangs then exits with message Killed
#362
Comments
Thinking a bit further about those numbers, it looks like |
Yes, I do not have any time to work on |
Thanks. Being a bit of a kludger, I might just write a wrapper script that takes my list of n maildir folders, and invokes |
When in doubt, brute force it. If it works, it's not a kludge. And machine time is cheaper than developer time. 😁 |
Still, the commit history of that project indicate there's a non-null chance of me refreshing the code base once a year. So if your patient you might see a new release of |
mdedup
hangs then exits with message Killed
@shirosaki just proposed PR #562 to reduce the memory usage of mail-deduplicate. I just merged it upstream and try to cut a release today. |
Just released mail-deduplicate 7.3.0, with performance enhancements from @shirosaki . I will close this issue for now on then. |
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Preliminary checks
Describe the bug
When running
mdedup
on a largeish maildir tree (tens of thousands of messages across dozens of folders, occupying about 3GB in total), the process gets about two-thirds of the way (by number of messages) through the "Compute hashes and group duplicates" phase, then hangs (at which point everything else running on the system also becomes very slow) for a few minutes, before exiting with the word "Killed" printed to the shell.I suspect that the hang results from
mdedup
's memory usage spiking to the point where lots of swap is being used, and the exit results frommdedup
being zapped by the OOM killer. Immediately before the hang,mdedup
's resident memory usage as reported bytop
is 1.8GB.To reproduce
Steps to reproduce the behavior:
The full
mdedup
CLI invocation you used.mdedup -s discard-all-but-one -a delete-discarded a_maildir_folder another_maildir_folder etcet era a_directory_which_as_well_as_being_a_maildir_folder_contains_subdirectories_which_are_also_maildir_folders
The data set leading to the bug.
Cannot provide this due to risk of confidential data leak (and the likelihood that the problem is specifically associated with the data set not being "minimal").
Expected behavior
mdedup
runs to completion.CLI output
Add here the raw copy of some console output you were able to produce. Some exemple includes:
Cannot provide this due to risk of confidential data leak.
Environment
All data on execution context as provided by
$ mdedup --version
:Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: