pixi performs poorly on high-latency filesystem

[kcajf]

Pixi is very fast on a local SSD.

When I'm trying to install on a mounted NFS filesystem (with a 100us + of latency), it really slows down a lot. e.g. on the "download & extract" progress line, I see speeds dropping down to 100s of KB/s (whereas on local SSD it's always 100MB/s+).
I'm not sure to what extent this is fundamental, and to what extent it comes down to e.g. some suboptimal IO buffering sizes and/or thread counts? Can anyone else reproduce?

[wolfv]

@kcajf yes, this has to do with how pixi currently downlaods & extracts files. It completely pipelines the download + extract, so that it directly writes the extracted contents.

I am thinking of changing that so that we first buffer to a temporary (or spooled temporary) file, and then extract from the file. This could make things more efficient for your use case and also keep download speeds high while extracting might take a little bit longer.

Do you think that could help in your case?

[kcajf]

Thanks for replying. Are you saying that IO (esp writing) is completely unbuffered? i.e. write() syscalls are for abitrarily small amounts of data? If that's true, then some buffering would probably help me a lot.

[kcajf]

hi @wolfv - did you have any further thoughts on this please?