Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unicode (utf-8) #33

Open
6 tasks
booniepepper opened this issue Jul 16, 2023 · 0 comments
Open
6 tasks

Unicode (utf-8) #33

booniepepper opened this issue Jul 16, 2023 · 0 comments
Milestone

Comments

@booniepepper
Copy link
Collaborator

booniepepper commented Jul 16, 2023

  • Choose a utf8 String implementation. (Promising: zigstr and utf8proc)
  • Implement it as the "String" type in dt.
    • Note: the dictionary type will have to move away from StringHashMap(Command) which not the worst thing ever, but also not trivial.
  • Expand tokenization to consider more kinds of whitespace (e.g. CJK full-width space)
  • Add runes and remove chars
  • Add more tests on string-sensitive commands like upcase, downcase, split, join, len, or eq
  • Add tests that include utf-8 source code (Like demos/世界を挨拶.dt)
@booniepepper booniepepper added this to the 2.0 milestone Jul 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant