Akka.Streams: FileIO sinks don't flush until completed

[Aaronontheweb]

Version: 1.4.17

The Akka.Streams FileIO sink stages have a design flaw:

akka.net/src/core/Akka.Streams/Implementation/IO/FileSubscriber.cs

Lines 104 to 112 in 444432a

	.With<OnNext>(next =>
	{
	try
	{
	var byteString = (ByteString) next.Element;
	var bytes = byteString.ToArray();
	_chan.Write(bytes, 0, bytes.Length);
	_bytesWritten += bytes.Length;
	}

Unlike the OutputStreamSubscriber, the FileSubscriber actor can't periodically flush data onto disk during the course of writing streaming output to a file - thus 100% of file content is retained in memory. This can be disastrous for applications writing any non-trivially large file.

We should, instead, use the OutputStreamSubscriber implementation and allow it to automatically flush periodically:

akka.net/src/core/Akka.Streams/Implementation/IO/OutputStreamSubscriber.cs

Line 40 in 444432a

Actor.Props.Create(() => new OutputStreamSubscriber(os, completionPromise, bufferSize, autoFlush))

We could also expose a command that could be injected from the outside to force flushing in accordance with specific domain rules specific to the user's application, but that might be a subject to a separate issue from this one.