Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize the lexer by iterating over bytes #335

Closed
Razican opened this issue Apr 18, 2020 · 2 comments · Fixed by #915
Closed

Optimize the lexer by iterating over bytes #335

Razican opened this issue Apr 18, 2020 · 2 comments · Fixed by #915
Assignees
@jevancc
Labels
E-Medium Medium difficulty problem Hacktoberfest Hacktoberfest 2021 - https://hacktoberfest.digitalocean.com lexer Issues surrounding the lexer performance Performance related changes and issues
Milestone
v0.11.0

Comments

@Razican
Copy link
Member

Razican commented Apr 18, 2020 •
edited
Loading

As maciejh mentioned in Reddit, iterating over characters is not fast. We should iterate over bytes, but we have to take into account that identifiers, strings, comments and more can have non-ASCII characters, since JavaScript is UTF-8.

Nonetheless, we can use byte-iteration for detecting language features much faster.

The comment:

Iterating over bytes rather than chars is inherently much faster, non-ascii UTF8 bytes are all >=128, and you might only really care about those being correct when parsing idents.
@Razican Razican added performance Performance related changes and issues lexer Issues surrounding the lexer labels Apr 18, 2020
@jasonwilliams jasonwilliams added this to the v0.10.0 milestone Jul 3, 2020
@jasonwilliams jasonwilliams modified the milestones: v0.10.0, v0.11.0 Sep 18, 2020
@jasonwilliams jasonwilliams added Hacktoberfest Hacktoberfest 2021 - https://hacktoberfest.digitalocean.com E-Medium Medium difficulty problem labels Oct 2, 2020
@jevancc
Copy link
Contributor

jevancc commented Oct 16, 2020 •
edited
Loading

Hi @Razican @jasonwilliams , I'd like to work on this issue. I can also look into how the new lexer can fix the utf-8 /utf-16 issues.

@jasonwilliams
Copy link
Member

Awesome @jevancc assigned

@jevancc jevancc mentioned this issue Oct 27, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jevancc jevancc

Labels
E-Medium Medium difficulty problem Hacktoberfest Hacktoberfest 2021 - https://hacktoberfest.digitalocean.com lexer Issues surrounding the lexer performance Performance related changes and issues
Projects
None yet
Milestone
v0.11.0
Development

Successfully merging a pull request may close this issue.

Improve lexer by make cursor iterate over bytes jevancc/boa
3 participants
@Razican @jasonwilliams @jevancc