Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn when running out of memory instead of segfaulting #251

Merged
merged 1 commit into from
Oct 3, 2016
Merged

Warn when running out of memory instead of segfaulting #251

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 30 additions & 4 deletions sambamba/sort.d
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,11 +121,23 @@ class Sorter {
_reads_capa = 1024;
auto sz = BamRead.sizeof * _reads_capa;
_reads = cast(BamRead*)std.c.stdlib.malloc(sz);
if (_reads is null) {
throw new Exception("alloc failed: no space for read pointers");
}
}

void clear() {
_used = 0;
_n_reads = 0;
if (_low_memory) {
auto realloc_storage = cast(ubyte*)std.c.stdlib.realloc(read_storage, max_sz / 2);
if (realloc_storage !is null) {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically, this realloc can also fail (could reallocate via allocate/copy/free). I choose not to throw an exception here, but to keep printing errors. This is a situation in which we can proceed, but sub-optimally, and cannot automatically do anything to improve things. Spamming the user probably highlights the problem but results should still be correct.

max_sz /= 2;
read_storage = realloc_storage;
stderr.writeln("reduced maximum buffer size to ", max_sz);
}
_low_memory = false;
}
}

void fill(R)(R* reads) {
Expand All @@ -135,11 +147,18 @@ class Sorter {
if (len + _used > max_sz)
break;

std.c.string.memcpy(read_storage + _used, read.raw_data.ptr, len);
if (_n_reads == _reads_capa) {
_reads_capa *= 2;
_reads = cast(BamRead*)std.c.stdlib.realloc(_reads, _reads_capa * BamRead.sizeof);
auto realloc_reads = cast(BamRead*)std.c.stdlib.realloc(_reads, 2 * _reads_capa * BamRead.sizeof);
if (realloc_reads is null) {
_low_memory = true;
stderr.writeln("realloc failed: system low on memory, limited to ", _reads_capa, " reads in buffer");
break;
} else {
_reads_capa *= 2;
_reads = realloc_reads;
}
}
std.c.string.memcpy(read_storage + _used, read.raw_data.ptr, len);
_reads[_n_reads].raw_data = read_storage[_used .. _used + len];
_reads[_n_reads].associateWithReader(read.reader);

Expand All @@ -154,7 +173,13 @@ class Sorter {
auto len = read.raw_data.length;
assert(len > max_sz);
_n_reads = 1;
read_storage = cast(ubyte*)std.c.stdlib.realloc(read_storage, len);
auto realloc_storage = cast(ubyte*)std.c.stdlib.realloc(read_storage, len);
if (realloc_storage is null) {
throw new Exception("realloc failed: not enough memory for read");
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are already trying to grow the storage, think there's nothing we can do here if it fails, so throw.

} else {
read_storage = realloc_storage;
max_sz = len;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

max_sz was not being updated previously, so the storage would grow but not get used past max_sz in future fills.

}
_used = len;
read_storage[0 .. len] = read.raw_data[];
_reads[0].raw_data = read_storage[0 .. _used];
Expand All @@ -175,6 +200,7 @@ class Sorter {

ubyte* read_storage;
size_t _used;
bool _low_memory;
}

static BamRead[] sortChunk(size_t n, BamRead[] chunk, TaskPool task_pool) {
Expand Down