Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: dynamically increase buffer size to handle processing large JSON lines #93

Merged
merged 2 commits into from
Jun 7, 2024

Conversation

noahgorstein
Copy link
Owner

@noahgorstein noahgorstein commented Jun 7, 2024

Per the bufio docs: https://pkg.go.dev/bufio#Scanner.Buffer:

Buffer sets the initial buffer to use when scanning and the maximum size of buffer that may be allocated during scanning. The maximum token size must be less than the larger of max and cap(buf). If max <= cap(buf), Scanner.Scan will use this buffer only and do no allocation.

By default, Scanner.Scan uses an internal buffer and sets the maximum token size to MaxScanTokenSize.
...

This became problematic for us when I introduced code to handle NDJSON (JSON lines) as input. If one of the lines was greater than 64KB (MaxScanTokenSize) or if the JSON was minified such that it was all on one line, we would run into an issue scanning each line because the maximum buffer used for reading would not have enough capacity.

This PR will still attempt to use a 64KB buffer to process each line but will keep retrying if the buffer is not large enough. Each retry will double the buffer size. The max buffer size is 100MB which is somewhat arbitrarily large but jqp will fail to be performant at this scale anyway due to syntax highlighting and writing large input/output to viewports so will keep it there for now.

@noahgorstein noahgorstein merged commit 2121bb4 into main Jun 7, 2024
2 checks passed
@noahgorstein noahgorstein deleted the fix-scanning-issues branch June 7, 2024 19:19
d-issy referenced this pull request in d-issy/dotfiles Jun 10, 2024
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [noahgorstein/jqp](https://github.com/noahgorstein/jqp) | minor |
`v0.6.0` -> `v0.7.0` |

---

### Release Notes

<details>
<summary>noahgorstein/jqp (noahgorstein/jqp)</summary>

###
[`v0.7.0`](https://github.com/noahgorstein/jqp/releases/tag/v0.7.0)

[Compare
Source](https://github.com/noahgorstein/jqp/compare/v0.6.0...v0.7.0)

#### Overview

Mostly small bug fixes and some performance improvements.

One new feature added is the ability to specify an optional query
argument to `jqp` cli that it will execute on startup.

curl "https://api.github.com/repos/jqlang/jq/issues" | jqp '.[] |
{"title": .title, "url": .url}'


https://github.com/noahgorstein/jqp/assets/23270779/735cf84a-dc6f-41e8-9659-5022a4937046


[@&#8203;EmilyGraceSeville7cf](https://github.com/EmilyGraceSeville7cf)
also added a JSON schema for jqp's config which should help users create
and edit their config files.

#### What's Changed

- typos suggestion by
[@&#8203;ccoVeille](https://github.com/ccoVeille) in
[https://github.com/noahgorstein/jqp/pull/59](https://github.com/noahgorstein/jqp/pull/59)
- code review by [@&#8203;ccoVeille](https://github.com/ccoVeille) in
[https://github.com/noahgorstein/jqp/pull/61](https://github.com/noahgorstein/jqp/pull/61)
- Add Continuous Integration to GitHub actions by
[@&#8203;ccoVeille](https://github.com/ccoVeille) in
[https://github.com/noahgorstein/jqp/pull/63](https://github.com/noahgorstein/jqp/pull/63)
- Add GitHub actions for checking typos by
[@&#8203;ccoVeille](https://github.com/ccoVeille) in
[https://github.com/noahgorstein/jqp/pull/62](https://github.com/noahgorstein/jqp/pull/62)
- Add dependabot to GitHub Action by
[@&#8203;ccoVeille](https://github.com/ccoVeille) in
[https://github.com/noahgorstein/jqp/pull/64](https://github.com/noahgorstein/jqp/pull/64)
- chore: address golangci-lint issues by
[@&#8203;noahgorstein](https://github.com/noahgorstein) in
[https://github.com/noahgorstein/jqp/pull/74](https://github.com/noahgorstein/jqp/pull/74)
- chore(deps): bump github.com/itchyny/gojq from 0.12.13 to 0.12.15 by
[@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/72](https://github.com/noahgorstein/jqp/pull/72)
- chore(deps): bump github.com/alecthomas/chroma/v2 from 2.12.0 to
2.13.0 by [@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/73](https://github.com/noahgorstein/jqp/pull/73)
- chore(deps): bump github.com/charmbracelet/bubbles from 0.16.1 to
0.18.0 by [@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/71](https://github.com/noahgorstein/jqp/pull/71)
- chore(deps): bump github.com/charmbracelet/lipgloss from 0.8.0 to
0.10.0 by [@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/70](https://github.com/noahgorstein/jqp/pull/70)
- chore(deps): bump github.com/spf13/cobra from 1.5.0 to 1.8.0 by
[@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/69](https://github.com/noahgorstein/jqp/pull/69)
- chore(deps): bump github.com/spf13/viper from 1.13.0 to 1.18.2 by
[@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/68](https://github.com/noahgorstein/jqp/pull/68)
- chore(deps): bump github.com/charmbracelet/bubbletea from 0.24.1 to
0.26.2 by [@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/67](https://github.com/noahgorstein/jqp/pull/67)
- chore(deps): bump actions/setup-go from 4 to 5 by
[@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/66](https://github.com/noahgorstein/jqp/pull/66)
- chore: upgrade to go v1.22 by
[@&#8203;noahgorstein](https://github.com/noahgorstein) in
[https://github.com/noahgorstein/jqp/pull/75](https://github.com/noahgorstein/jqp/pull/75)
- refactor: reduce complexity of various methods/functions by
[@&#8203;noahgorstein](https://github.com/noahgorstein) in
[https://github.com/noahgorstein/jqp/pull/76](https://github.com/noahgorstein/jqp/pull/76)
- feat: add optional cli argument to specify initial query by
[@&#8203;noahgorstein](https://github.com/noahgorstein) in
[https://github.com/noahgorstein/jqp/pull/77](https://github.com/noahgorstein/jqp/pull/77)
- bug: dont set viewport content on resize by
[@&#8203;noahgorstein](https://github.com/noahgorstein) in
[https://github.com/noahgorstein/jqp/pull/79](https://github.com/noahgorstein/jqp/pull/79)
- chore(deps): bump github.com/itchyny/gojq from 0.12.15 to 0.12.16 by
[@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/85](https://github.com/noahgorstein/jqp/pull/85)
- chore(deps): bump github.com/alecthomas/chroma/v2 from 2.13.0 to
2.14.0 by [@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/80](https://github.com/noahgorstein/jqp/pull/80)
- chore(deps): bump github.com/charmbracelet/lipgloss from 0.10.0 to
0.11.0 by [@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/82](https://github.com/noahgorstein/jqp/pull/82)
- chore(deps): bump github.com/spf13/viper from 1.18.2 to 1.19.0 by
[@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/84](https://github.com/noahgorstein/jqp/pull/84)
- chore(deps): bump github.com/charmbracelet/bubbletea from 0.26.2 to
0.26.4 by [@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/83](https://github.com/noahgorstein/jqp/pull/83)
- fix: revert to lipgloss v0.10.0 temporarily by
[@&#8203;noahgorstein](https://github.com/noahgorstein) in
[https://github.com/noahgorstein/jqp/pull/87](https://github.com/noahgorstein/jqp/pull/87)
- fix: don't block while inputdata bubble initializes by
[@&#8203;noahgorstein](https://github.com/noahgorstein) in
[https://github.com/noahgorstein/jqp/pull/86](https://github.com/noahgorstein/jqp/pull/86)
- fix: nil pointer dereference as a result of accessing state before
queryinput initialized by
[@&#8203;noahgorstein](https://github.com/noahgorstein) in
[https://github.com/noahgorstein/jqp/pull/90](https://github.com/noahgorstein/jqp/pull/90)
- fix: dynamically increase buffer size to handle processing large JSON
lines by [@&#8203;noahgorstein](https://github.com/noahgorstein) in
[https://github.com/noahgorstein/jqp/pull/93](https://github.com/noahgorstein/jqp/pull/93)
- chore(deps): bump github.com/charmbracelet/lipgloss from 0.10.0 to
0.11.0 by [@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/89](https://github.com/noahgorstein/jqp/pull/89)
- chore(deps): bump crate-ci/typos from 1.21.0 to 1.22.1 by
[@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/noahgorstein/jqp/pull/92](https://github.com/noahgorstein/jqp/pull/92)
- feat: implement json schema for config by
[@&#8203;EmilyGraceSeville7cf](https://github.com/EmilyGraceSeville7cf)
in
[https://github.com/noahgorstein/jqp/pull/78](https://github.com/noahgorstein/jqp/pull/78)
- chore: prep v0.7.0 release by
[@&#8203;noahgorstein](https://github.com/noahgorstein) in
[https://github.com/noahgorstein/jqp/pull/94](https://github.com/noahgorstein/jqp/pull/94)

#### New Contributors

- [@&#8203;ccoVeille](https://github.com/ccoVeille) made their first
contribution in
[https://github.com/noahgorstein/jqp/pull/59](https://github.com/noahgorstein/jqp/pull/59)
- [@&#8203;dependabot](https://github.com/dependabot) made their first
contribution in
[https://github.com/noahgorstein/jqp/pull/72](https://github.com/noahgorstein/jqp/pull/72)
-
[@&#8203;EmilyGraceSeville7cf](https://github.com/EmilyGraceSeville7cf)
made their first contribution in
[https://github.com/noahgorstein/jqp/pull/78](https://github.com/noahgorstein/jqp/pull/78)

**Full Changelog**:
noahgorstein/jqp@v0.6.0...v0.7.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/d-issy/dotfiles).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTMuMCIsInVwZGF0ZWRJblZlciI6IjM3LjM5My4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant