Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Long filename contains unicode characters with by-file truncates error #214

Closed
lingsamuel opened this issue Dec 23, 2020 · 8 comments
Closed

Long filename contains unicode characters with by-file truncates error #214

lingsamuel opened this issue Dec 23, 2020 · 8 comments
Labels
bug Something isn't working

Comments

@lingsamuel
Copy link

lingsamuel commented Dec 23, 2020 •
edited
Loading

Describe the bug
Long filename contains unicode characters with by-file truncated incorrectly.

To Reproduce

touch "ббббббббббббббббббббббббббббббббб.md"
touch "ѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬ.md"
touch "中文中文中文中文中文中文中文中文中文中文中文中文中文中文中文中文.md"
scc --by-file

Output:

image

Expected behavior
�� shouldn't appear, indent should works properly.

Desktop (please complete the following information):

  • OS: Linux 5.9.9-arch1-1 (Arch Linux)
  • scc version 2.13.0, make from source
@lingsamuel lingsamuel changed the title Long filename contains double-byte and multi-byte characters with by-file truncates error Long filename contains unicode characters with by-file truncates error Dec 23, 2020
@lingsamuel
Copy link
Author

б https://www.compart.com/en/unicode/U+0431
Ѭ https://www.compart.com/en/unicode/U+046C
中 https://www.compart.com/en/unicode/U+4E2D

@boyter
Copy link
Owner

boyter commented Dec 24, 2020

You would think I would be aware of this considering how much time I have spent looking over unicode recently.

Its down to it using bytes and not characters. Its especially annoying because it trims from the front, which isn't how Go lets you walk runes.... but not impossible to rectify. I can see the size never matching up though depending on the characters though, but we can fix the current issues for sure.

@boyter boyter added the bug Something isn't working label Dec 24, 2020
boyter added a commit that referenced this issue Jan 4, 2021
@boyter
partial fix for #214
b35e0e1
boyter added a commit that referenced this issue Jan 4, 2021
@boyter
Merge pull request #218 from boyter/issue214
a69370e 
partial fix for #214
@boyter
Copy link
Owner

boyter commented Jan 4, 2021 •
edited
Loading

Done. Note there isn't much I can do about 中 which for some reason does not display in mono-spaced font... tokei has the same issue. However at least its not cut off anymore when printing.

If its urgent, a build from master should resolve this for you :)

image

@boyter boyter closed this as completed Jan 4, 2021
@dbaggerman
Copy link
Collaborator

Note there isn't much I can do about 中 which for some reason does not display in mono-spaced font

You could have a play around with https://godoc.org/golang.org/x/text/width to calculate the display width.

@boyter
Copy link
Owner

boyter commented Jan 5, 2021

I actually found this just then http://github.com/mattn/go-runewidth which might be able to solve this.

@boyter boyter reopened this Jan 5, 2021
@boyter
Copy link
Owner

boyter commented Jan 5, 2021 •
edited
Loading

Added the above in... and its all there.. however it appears sprintf has issues with this as well... but moved over to some custom parsing. Anyway merged in to master.

image

@boyter boyter closed this as completed Jan 6, 2021
@lingsamuel
Copy link
Author

Thanks!

@boyter
Copy link
Owner

boyter commented Jan 10, 2021

@lingsamuel No problem at all.

lukas-brenning added a commit to lukas-brenning/scc that referenced this issue Jan 13, 2021
@lukas-brenning @boyter
@AlDanial @arp242
Test (#1)
93de68b 
* Update README.md

* SLOCCount came before cloc

* resolve boyter#206

* First cut at boyter#210

* Add "gohtml" and "gotxt" to the list of extensions for Go templates

These are the default extensions that e.g. GoLand uses, and some other
tools as well I believe. It's useful as it disambiguates between
templates for html/template and text/template.

* partial fix for boyter#214

* resolve after feedback

* Mostly ready for boyter#211

* real fix for issue214

* few tests and typo

* Resolvehttps://github.com/boyter/issues/208

* Update to remove wrong Go version mentioned boyter#220

Co-authored-by: Ben Boyter <boyter@users.noreply.github.com>
Co-authored-by: AlDanial <al.danial@gmail.com>
Co-authored-by: Ben Boyter <ben@boyter.org>
Co-authored-by: Martin Tournoij <martin@arp242.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@boyter @dbaggerman @lingsamuel