Skip to content

Commit

Permalink
Buffer zngio.scanner.workerCh for more concurrency (#5103)
Browse files Browse the repository at this point in the history
Tracing with package runtime/trace shows the scanner.start and
worker.run goroutines block frequently on the workerCh channel.  Add
buffering to workerCH to reduce blocking and increase concurrency.

When scanning a 4 GB ZNG file containing Zeek logs, this yields a 1.1X
speedup on my 10-core machine.
  • Loading branch information
nwt authored Apr 5, 2024
1 parent e6ce847 commit d3fef6d
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion zio/zngio/scanner.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ func newScanner(ctx context.Context, zctx *zed.Context, r io.Reader, filter zbuf
maxSize: opts.Max,
},
validate: opts.Validate,
workerCh: make(chan *worker),
workerCh: make(chan *worker, opts.Threads+1),
resultChCh: make(chan chan op.Result, opts.Threads+1),
}
for i := 0; i < opts.Threads; i++ {
Expand Down

0 comments on commit d3fef6d

Please sign in to comment.