Chunking/streaming option for final database output?

[ccbaumler]

An ongoing issue I am having is the interruption of the download for extremely large databases (i.e. ~1,000,000 bacterial genomes). This requires me to restart the download from 0%.

Is it possible to stream the downloads into the output file and check for an existing output file?

This could have the additional benefit of trying to re-capture the failures as well.

@bluegenes states that this is currently not possible, but suggests to batch the output and collect the batched files into a single database.