Skip to content

Commit

Permalink
pq: eliminate corruption by forcing version byte to be persisted
Browse files Browse the repository at this point in the history
When the PQ creates a new page and allocates a memory-mapped buffer, the
underlying file is zero'd out to full page capacity and the version byte is
written to the buffer.

If Logstash crashes or is shut down before any elements have been pushed into
the queue page, we have no guarantees that the version marker has been
persisted to the storage device. A subsequent attempt to load an all-zeros
queue page will result in an obscure error message and failure to load:

~~~
AbstractPipelineExt - Logstash failed to create queue.
org.logstash.ackedqueue.io.MmapPageIOV2$PageIOInvalidVersionException: Expected page version=2 but found version=0
~~~

By sending `MappedByteBuffer#force()` immediately after the version has been
added to the buffer, we can shrink the window in which a crash can leave the
queue on disk in a corrupt state.
  • Loading branch information
yaauie authored and elasticsearch-bot committed Jan 11, 2021
1 parent 3faa83f commit 622c510
Showing 1 changed file with 1 addition and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,7 @@ public void create() throws IOException {
}
buffer.position(0);
buffer.put(VERSION_TWO);
buffer.force();
this.head = 1;
this.minSeqNum = 0L;
this.elementCount = 0;
Expand Down

0 comments on commit 622c510

Please sign in to comment.