Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic on orig_bytes pool key #2960

Closed
mccanne opened this issue Aug 26, 2021 · 1 comment · Fixed by #2969
Closed

panic on orig_bytes pool key #2960

mccanne opened this issue Aug 26, 2021 · 1 comment · Fixed by #2969
Assignees
Labels
bug Something isn't working
Milestone

Comments

@mccanne
Copy link
Collaborator

mccanne commented Aug 26, 2021

@aousterh found this bug experimenting with different pool keys.

Reproduce with:

zed lake create -p bytes -orderby orig_bytes
zed lake load -p bytes conn.log.gz
zed lake query -z "from bytes range 20 to 30 | count()"

gives

fatal error: concurrent map writes
fatal error: concurrent map writes

goroutine 73 [running]:
runtime.throw(0x1c3b6b6, 0x15)
	/usr/local/go/src/runtime/panic.go:1117 +0x72 fp=0xc0005bb898 sp=0xc0005bb868 pc=0x10398b2
runtime.mapassign(0x1af67a0, 0xc000359ec0, 0xc0005bba38, 0x2429640)
	/usr/local/go/src/runtime/map.go:676 +0x591 fp=0xc0005bb918 sp=0xc0005bb898 pc=0x1011871
github.com/brimdata/zed/expr.compareValues(0x1e066f0, 0x2427b60, 0xc0009ea040, 0x1, 0x3fc0, 0x1e066f0, 0x2427b60, 0xc0003fa093, 0x0, 0x7ff6d, ...)
...
@nwt nwt self-assigned this Aug 26, 2021
@nwt nwt added this to the Data MVP0 milestone Aug 26, 2021
@nwt nwt added the bug Something isn't working label Aug 26, 2021
@nwt nwt closed this as completed in #2969 Aug 27, 2021
brim-bot pushed a commit to brimdata/brimcap that referenced this issue Aug 27, 2021
…nt use" by nwt

This is an auto-generated commit with a Zed dependency update. The Zed PR
brimdata/super#2969, authored by @nwt,
has been merged.

lake: make rangeWrapper filters safe for concurrent use

Because of concurrency in the ZNG scanner, separate filters returned by
a zbuf.Filter's methods must be safe for concurrent use in separate
goroutines.  Filters returned by lake.rangeWrapper.AsFilter do not have
this property because they share the single expr.ValueCompareFn in
rangeWrapper.compare.  Fix this by creating a new expr.ValueCompareFn
for each filter returned.

Closes brimdata/super#2960.
brim-bot pushed a commit to brimdata/brimcap that referenced this issue Aug 27, 2021
…nt use" by nwt

This is an auto-generated commit with a Zed dependency update. The Zed PR
brimdata/super#2969, authored by @nwt,
has been merged.

lake: make rangeWrapper filters safe for concurrent use

Because of concurrency in the ZNG scanner, separate filters returned by
a zbuf.Filter's methods must be safe for concurrent use in separate
goroutines.  Filters returned by lake.rangeWrapper.AsFilter do not have
this property because they share the single expr.ValueCompareFn in
rangeWrapper.compare.  Fix this by creating a new expr.ValueCompareFn
for each filter returned.

Closes brimdata/super#2960.
brim-bot pushed a commit to brimdata/zui that referenced this issue Aug 27, 2021
…nt use" by nwt

This is an auto-generated commit with a Zed dependency update. The Zed PR
brimdata/super#2969, authored by @nwt,
has been merged.

lake: make rangeWrapper filters safe for concurrent use

Because of concurrency in the ZNG scanner, separate filters returned by
a zbuf.Filter's methods must be safe for concurrent use in separate
goroutines.  Filters returned by lake.rangeWrapper.AsFilter do not have
this property because they share the single expr.ValueCompareFn in
rangeWrapper.compare.  Fix this by creating a new expr.ValueCompareFn
for each filter returned.

Closes brimdata/super#2960.
@philrz
Copy link
Contributor

philrz commented Aug 27, 2021

I've verified in Zed commit a2b7592 that this no longer causing a panic.

$ zed -version
Version: v0.29.0-479-ga2b7592f

$ zed lake init
using environment variable ZED_LAKE_ROOT
lake created: /Users/phil/logs

$ zed lake create -p bytes -orderby orig_bytes
pool created: bytes

$ zed lake load -p bytes conn.log.gz
1xKF04dJylry7AoeLro4jieLFBh committed

$ zed lake query -z "from bytes range 20 to 30 | count()"
[no output]

$ echo $?
0

However, revisiting the spirit of what was originally being attempted, it does seem like this should have produced some output, since I can see values of the pool key in the specified range.

$ zed lake query -z "from bytes | count() by orig_bytes | sort orig_bytes | head 31"
{orig_bytes:0(uint64),count:915494(uint64)}(=0)
{orig_bytes:1,count:171}(0)
{orig_bytes:2,count:2}(0)
{orig_bytes:3,count:200}(0)
{orig_bytes:4,count:101}(0)
{orig_bytes:5,count:8}(0)
{orig_bytes:6,count:235}(0)
{orig_bytes:7,count:26}(0)
{orig_bytes:8,count:48}(0)
{orig_bytes:9,count:87}(0)
{orig_bytes:10,count:12}(0)
{orig_bytes:11,count:96}(0)
{orig_bytes:12,count:82}(0)
{orig_bytes:13,count:3}(0)
{orig_bytes:14,count:28}(0)
{orig_bytes:15,count:23}(0)
{orig_bytes:16,count:45}(0)
{orig_bytes:17,count:15}(0)
{orig_bytes:18,count:69}(0)
{orig_bytes:19,count:64}(0)
{orig_bytes:20,count:34}(0)
{orig_bytes:21,count:22}(0)
{orig_bytes:22,count:48}(0)
{orig_bytes:23,count:36}(0)
{orig_bytes:24,count:253}(0)
{orig_bytes:25,count:248}(0)
{orig_bytes:26,count:26}(0)
{orig_bytes:27,count:1}(0)
{orig_bytes:28,count:14}(0)
{orig_bytes:29,count:11}(0)
{orig_bytes:30,count:9}(0)

I've opened separate issue #2970 to track this.

Thanks @nwt!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants