-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Old-style CR terminators can affect classification #2988
Comments
This issue has been automatically marked as stale because it has not had activity in a long time. If this issue is still relevant and should remain open, please reply with a short explanation (e.g. "I have checked the code and this issue is still relevant because ___."). Thank you for your contributions. |
. |
Removing the Stale label is also enough to indicate activity and stalebot will ignore the issue/PR for another 30 days. |
Oh. 😢 Sorry. |
This issue has been automatically marked as stale because it has not had activity in a long time. If this issue is still relevant and should remain open, please reply with a short explanation (e.g. "I have checked the code and this issue is still relevant because ___."). Thank you for your contributions. |
This issue has been automatically marked as stale because it has not had activity in a long time. If this issue is still relevant and should remain open, please reply with a short explanation (e.g. "I have checked the code and this issue is still relevant because ___."). Thank you for your contributions. |
Alright, so it's not the CR line-terminators that's screwing up classification. If I rerun In text, the In any case, a few new samples and a carefully-written heuristic should put this matter to rest. 👍 EDIT: If it's worth mentioning, I did scour the TeX samples searching for any stray carriage-returns (which weren't locked into a never-ending partnership with a |
Also, it's kind of strange that this offending fixture has been sitting in Linguist's index a71f827..ec9ba7a 100644
--- /samples/World of Warcraft Addon Data/ChatKeys-CR.toc
+++ /samples/World of Warcraft Addon Data/ChatKeys-LF.toc
@@ -1,16 +1,3 @@
- ChatKeys-CR.toc
- 10 lines (9 sloc)
- type: Text
- mime type: text/plain
- language: TeX <~~~~~~~~ Wrong!
- ^^^^^
@@ -67,13 +54,15 @@
+ ChatKeys-LF.toc
+ (10 lines, 9 sloc)
+ type: Text
+ mime type: text/plain
+ language: World of Warcraft Addon Data
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ Correct! Adding them both and writing a test to verify they pass as WoWToC data correctly. Will have a look at giving that TeX confusion a few heuristics... |
Seems as though the heuristics weren't working because However, if I remove the heuristic altogether and rerun Linguist, I get the same results as before anyway, indicating that CR terminators are affecting both Linguist's classifier and its heuristics. I'm not sure if this issue is even worth considering, actually. I mean, the only system that used |
Closing this issue as our upcoming new classifier, courtesy of @smola's heroic efforts, indirectly fixes this issue. |
This was discovered in #2971, where one file was still being misclassified because of its unusual line-endings.
With support for WoW-ToC files added,
ChatKeys.toc
should've been identified as addon metadata. However, Linguist still interpreted it as TeX until the line endings were converted from old-style CR to proper modern LF. Suddenly, it identified as WoW-ToC again.Could choice of line-ending be potentially skewing the classification?
Converting the file to LF endings wasn't possible in Atom due to a bug (which was recently addressed), so I had to use Perl. I've attached the file with both line-ending types to spare you the trouble.
The text was updated successfully, but these errors were encountered: