crashes: Cannot allocate memory

[burgalon]

2016-05-21T22:31:48.042-0400 E STORAGE  [thread1] WiredTiger (12) [1463884307:558146][277593:0x7faf94a62700], file:collection-2--8597162973160007227.wt, WT_SESSION.checkpoint: collection-2--8597162973160007227.wt write error: failed to write 8192 bytes at offset 140288000: Cannot allocate memory
2016-05-21T22:31:48.048-0400 E STORAGE  [thread1] WiredTiger (12) [1463884308:48790][277593:0x7faf94a62700], checkpoint-server: checkpoint server error: Cannot allocate memory
2016-05-21T22:31:48.048-0400 E STORAGE  [thread1] WiredTiger (-31804) [1463884308:48832][277593:0x7faf94a62700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-05-21T22:31:48.048-0400 I -        [thread1] Fatal Assertion 28558
2016-05-21T22:31:48.048-0400 I -        [thread1] 
***aborting after fassert() failure
2016-05-21T22:31:48.053-0400 I -        [WTJournalFlusher] Fatal Assertion 28559
2016-05-21T22:31:48.053-0400 I -        [WTJournalFlusher] 
***aborting after fassert() failure
2016-05-21T22:31:48.085-0400 I -        [conn12] Fatal Assertion 28559
2016-05-21T22:31:48.085-0400 I -        [conn12] 
***aborting after fassert() failure
2016-05-21T22:31:48.246-0400 F -        [thread1] Got signal: 6 (Aborted).
 0x12d6632 0x12d5559 0x12d5d62 0x3000c0f7e0 0x3000432625 0x3000433e05 0x126cb32 0x1063543 0x1a2935c 0x1a294fd 0x1a298e4 0x19b2a4b 0x3000c07aa1 0x30004e893d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"ED6632","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"ED5559"},{"b":"400000","o":"ED5D62"},{"b":"3000C00000","o":"F7E0"},{"b":"3000400000","o":"32625","s":"gsignal"},{"b":"3000400000","o":"33E05","s":"abort"},{"b":"400000","o":"E6CB32","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C63543"},{"b":"400000","o":"162935C","s":"__wt_eventv"},{"b":"400000","o":"16294FD","s":"__wt_err"},{"b":"400000","o":"16298E4","s":"__wt_panic"},{"b":"400000","o":"15B2A4B"},{"b":"3000C00000","o":"7AA1"},{"b":"3000400000","o":"E893D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-573.12.1.el6.x86_64", "version" : "#1 SMP Mon Nov 23 12:55:32 EST 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFEE26DF000", "elfType" : 3 }, { "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3 }, { "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12d6632]
 mongod(+0xED5559) [0x12d5559]
 mongod(+0xED5D62) [0x12d5d62]
 libpthread.so.0(+0xF7E0) [0x3000c0f7e0]
 libc.so.6(gsignal+0x35) [0x3000432625]
 libc.so.6(abort+0x175) [0x3000433e05]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x126cb32]
 mongod(+0xC63543) [0x1063543]
 mongod(__wt_eventv+0x40C) [0x1a2935c]
 mongod(__wt_err+0x8D) [0x1a294fd]
 mongod(__wt_panic+0x24) [0x1a298e4]
 mongod(+0x15B2A4B) [0x19b2a4b]
 libpthread.so.0(+0x7AA1) [0x3000c07aa1]
 libc.so.6(clone+0x6D) [0x30004e893d]
-----  END BACKTRACE  -----

[icflorescu]

Context?

[burgalon]

Small gear... I had a cron job that has lots of queries and inserts and seems to crash the mongo...
Any other details in mind?

[burgalon]

Seems like we need to set storage.wiredTiger.engineConfig.cacheSizeGB

[burgalon]

From what it seems there is no way to run MongoDB properly on a small instance. You'd need a medium with 1GB min

[icflorescu]

Thanks a lot for taking the time to really look into this!

I've dug a bit myself after you've raised the issue and learned that other people ran into the same on various occasions. I also found a potentially interesting discussion here: http://comments.gmane.org/gmane.comp.db.mongodb.user/122126

Apparently, we could set a lower limit with --wiredTigerEngineConfigString="cache_size=xxxM", although the official documentation says 1GB will be the minimum.

I'll try setting it to 256MB to accommodate the default 512MB limit on OpenShift and see what happens, but if your data is important, I'd highly suggest you to switch to hosting your database on a specialized provider such as mLab.

There's also the option of switching to MMAPv1, but that would completely negate the benefits of WiredTiger's granular concurrency controls (basically MMAPv1 locks at the collection level while WiredTiger only locks at the document level).

[icflorescu]

Going to close this for now, since there's not much we can do about it besides what I already did in this commit.

Feel free to reopen if you have a better idea.