Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes needed for scanning of large files #216

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tomlea
Copy link

@tomlea tomlea commented Nov 5, 2021

Fixes to allow my use case of parsing 500mb+ file, with huge blocks of content that do not need parsing, but was taking forever to do without.

  • Reparse for error logging no longer resets to 0, it returns to the starting position. This is helpful for parsing lots of entries from a stream.
  • Adds Source#consume_until which allows me to scan large blocks of code looking for an end symbol.

@kschiess
Copy link
Owner

kschiess commented Dec 4, 2022

Sounds like a bit of a new use case for parslet; new in the sense that it was never going to work on huge inputs and wasn't made for it - parsing is just too slow. In your case: Would it be impossible to extract parseable bits by other means and then parse the remaining important parts with parslet?

If I merge and then announce parslet works with 500's of MBs, I am putting us in a bad spot...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants