Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unicode file names in git-log are escaped #42

Open
rimutaka opened this issue Nov 2, 2021 · 2 comments
Open

Unicode file names in git-log are escaped #42

rimutaka opened this issue Nov 2, 2021 · 2 comments
Labels
bug Something isn't working

Comments

@rimutaka
Copy link
Member

rimutaka commented Nov 2, 2021 •
edited
Loading

A file name with non-ascii chars get encoded and escaped in git-log making it impossible to parse. This purely git-log issue. The blobs can be accessed with no issues.

  • File name: til/LINQサンプル.cs/LINQサンプル.cs/Program.cs
  • Git log: "LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/Program.cs"
  • STM report: "\"CsharpRxSample/CsharpRxSample/IObserver_T_\\343\\201\\256\\347\\234\\201\\347\\225\\245/_Observer.cs\""
  • repo: https://github.com/184ym2/til

It's not clear how to convert the numbers back to UTF-8. E.g.

  • Text (4 glyphs): サンプル
  • Encoded in log(12 numbers): \343\202\265\343\203\263\343\203\227\343\203\253
  • Text as bytes (12 bytes): 227 130 181 227 131 179 227 131 151 227 131 171

Resources

  1. https://git-scm.com/docs/git-log#Documentation/git-log.txt---encodingltencodinggt
  2. https://stackoverflow.com/questions/69817372/how-to-decode-escaped-file-names-in-git-log
  3. https://en.wikipedia.org/wiki/Escape_sequences_in_C#Table_of_escape_sequences
  4. https://git-scm.com/docs/git-config#Documentation/git-config.txt-corequotePath

Git log dump

commit a2ed0ef590f9b04d6e548a522a0be75053c2eb4a (HEAD -> master, origin/master, origin/HEAD)
Author: ymhome <60121246+184ym2@users.noreply.github.com>
Date:   Tue Mar 16 05:33:30 2021 +0900

    20210316

"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/.vs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/v16/.suo"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ/Aggregation.cs"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ/Sort.cs"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.csproj"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/Program.cs"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/bin/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.exe"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/bin/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.pdb"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/obj/x86/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.csproj.CoreCompileInputs.cache"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/obj/x86/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.csprojAssemblyReference.cache"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/obj/x86/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.exe"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/obj/x86/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.pdb"

commit a6674fc5ed1ad277ea1325386cacd58d82364057
Merge: 2a11501 fd1a5aa
Author: ymhome <60121246+184ym2@users.noreply.github.com>
Date:   Thu Mar 11 20:51:48 2021 +0900

    Merge branch 'master' of github.com:184ym2/til into master

commit 2a1150123979283b5d4f6048be04306f6c3a3a56
Author: ymhome <60121246+184ym2@users.noreply.github.com>
Date:   Thu Mar 11 20:48:46 2021 +0900

    20210311

"CsharpObject\343\201\256\350\244\207\350\243\275Sample/.vs/CsharpObject\343\201\256\350\244\207\350\243\275Sample/v16/.suo"
"CsharpObject\343\201\256\350\244\207\350\243\275Sample/CsharpObject\343\201\256\350\244\207\350\243\275Sample/obj/x86/Debug/CsharpObject\343\201\256\350\244\207\350\243\275Sample.csproj.CoreCompileInputs.cache"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/.vs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/v16/.suo"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ/Evaluation.cs"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ/Projection.cs"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ/Sort.cs"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.csproj"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/Program.cs"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/TeamList.cs"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/bin/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.exe"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/bin/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.pdb"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/obj/x86/Debug/.NETFramework,Version=v4.0,Profile=Client.AssemblyAttributes.cs"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/obj/x86/Debug/DesignTimeResolveAssemblyReferencesInput.cache"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/obj/x86/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.csproj.CoreCompileInputs.cache"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/obj/x86/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.csproj.FileListAbsolute.txt"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/obj/x86/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.csprojAssemblyReference.cache"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/obj/x86/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.exe"
"LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs/obj/x86/Debug/LINQ\343\202\265\343\203\263\343\203\227\343\203\253.cs.pdb"
@rimutaka rimutaka added the bug Something isn't working label Nov 2, 2021
rimutaka added a commit that referenced this issue Nov 3, 2021
@rimutaka
Ignoring file names encoded by git #42
22751b1 
* a temp fix to ignore any file names or log entries enclosed in " "
@rimutaka
Copy link
Member Author

rimutaka commented Nov 3, 2021

A possible solution can be to use https://docs.rs/combine/4.6.2/combine/parser/repeat/fn.escaped.html or https://github.com/Geal/nom

@rimutaka
Copy link
Member Author

rimutaka commented Nov 3, 2021

Git -z option removes the encoding, but that produces a very messy output that is even harder to parse.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rimutaka