bufio scanner token too long

[SoftTools59654]

I encountered the following error while importing data in line
bufio scanner token too long
The problem is related to the 64kb limit

[philrz]

@SoftTools59654: I've moved this to the Zed repo since it's something we'd have to address first at that level.

Here's a repro with Zed commit 4dc6236.

$ zq -version
Version: v1.14.0-2-g4dc62369

$ perl -E 'say "=" x 65536' | zq -i line 'count()' -
stdio:stdin: bufio.Scanner: token too long

Whereas it works with one character less.

$ perl -E 'say "=" x 65535' | zq -i line 'count()' -
1(uint64)

While it seems there's likely to always be some upper limit on the size of values, upon seeing this @mccanne remarked that we should indeed be able to increase the current limit to something measured in megabytes.

[philrz]

Verified in Zed commit f9325b7.

Per the attached PRs, the buffer that was formerly 64 KB has now been increased to 25 MB. Therefore the crossover point is now:

$ zq -version
Version: v1.14.0-4-gf9325b72

$ perl -E 'say "=" x 26214399' | zq -i line 'count()' -
1(uint64)

$ perl -E 'say "=" x 26214400' | zq -i line 'count()' -
stdio:stdin: bufio.Scanner: token too long

Thanks @mattnibs!

[philrz]

@SoftTools59654: I saw you clicked a 👍 on the last comment so it looks like you're already aware this buffer has been increased at the Zed layer. I'm not sure if you're a Zui Insiders user, but FYI the latest Zui Insiders release 1.6.1-13 does include the newer Zed so you can start loading bigger values with that before the next set of GA releases comes out. I also wanted to make you aware that testing the effects of the Zed change in Zui revealed a separate limit at the Zui layer where values bigger than about 2 MB seem to be currently rendered blank, so issue brimdata/zui#3016 will be used to track addressing that.