New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Benchmark Demo Performance Improvements #193

Open

stanbrub opened this issue Oct 19, 2023 · 0 comments

Labels

Collaborator

stanbrub commented Oct 19, 2023 •

edited

Loading

Currently, when the Benchmark Demo is run, it takes about 20 seconds. That's too long for a demo.
Issues:

The demo server is Google. The NFS server is Google. The pipe between them appears to be slow. (Throttling?)
The tables are loaded one-by-one rather than in parallel
Load the CSV the files in parallel, at least for the same table
Compress the data in GCloud Gzip CSV for Benchmark Cmd Line Results #177

Notes:

On the demo server, reading csv.gz vs csv does not improve demo performance. NFS appears to be the problem.
If you run the demo when the worker first spins up, the demo takes 20 sec. Run it subsequently, and it takes 2-3 secs.

stanbrub added the enhancement label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment