You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Reading 32bytes per 32bytes a 6GB list of points from a os.File is terrible for perf.
Without changing the code, just wrapping the os.Fileio.Reader in to a bufio.Reader (for example bufio.NewReaderSize(f, fr.Bytes*(1<<10))) improves performance by a 10x factor.
Similar logic for io.Writer and bufio.Writer .
However our APIs work with Writer and Reader, and doing a bufio.Reader inside the functions may result in the function reading too many bytes. (calls to read functions are sometime encapsulated or following one another with the same Reader) .
Maybe we could add a high level ReadFromFile to avoid putting the burden of wrapping the reader on the caller;
The text was updated successfully, but these errors were encountered:
Reading 32bytes per 32bytes a 6GB list of points from a
os.File
is terrible for perf.Without changing the code, just wrapping the
os.File
io.Reader
in to abufio.Reader
(for examplebufio.NewReaderSize(f, fr.Bytes*(1<<10))
) improves performance by a 10x factor.Similar logic for
io.Writer
andbufio.Writer
.However our APIs work with
Writer
andReader
, and doing abufio.Reader
inside the functions may result in the function reading too many bytes. (calls to read functions are sometime encapsulated or following one another with the sameReader
) .Maybe we could add a high level
ReadFromFile
to avoid putting the burden of wrapping the reader on the caller;The text was updated successfully, but these errors were encountered: