Switch to gengo

[spenserblack]

See #1152 for previous conversation. With the drastic API changes to gengo it started to feel like more work to modify the existing PR instead of starting over.

To Do

• Gengo needs to support the remaining languages (or at least have reasons to not support them) (Onefetch languages checklist spenserblack/gengo#34)
• Change pretty much everything that previously matched on Language and tokei::LanguageType enums.
• Drop the "blob size" stat (Switch to gengo #1305 (comment))

Resolves #26
Closes #1152

[spenserblack]

@o2sh Before this moves any further we need to determine if we want to move forward with analyzing a git ref (HEAD) or a folder (I think preferably we don't support both). I've started listing reasons for each here: #1303

[o2sh]

It would be nice to have language type and glob filtering for the V1

[o2sh]

or is this going to be handled via .gitattributes?

[spenserblack]

That depends on the "file source." Right now the Git file source is highly opinionated and would use .gitattributes for ignoring files, overrides, etc.

---

I plan on the Directory file source being much less opinionated (can't get much more generic than just reading a folder 🙂). The functionality isn't there yet, but I was considering adding ways to configure excluding/including files. Right now .gitignore and .ignore files would ignore files via the ignore crate's default behavior. I was hoping to discuss the usage sometime, but it would probably either be like this:

let file_source = Directory::with_config("./", Config { ignored_files: Some(vec![]) });

or like this:

let file_source = Directory::new("./").extend_ignored_files(&[]);

[o2sh]

Very clear, and it also aplies to the language type filter ?

[spenserblack]

By default Gengo marks Programming, Markup, and Query as detectable, and Prose and Data as not detectable. detectable is basically a boolean that answers "should this factor in to the stats?" Also, documentation, generated files, and vendored files are not detectable by default. Assuming we use the Git file source with .gitattributes, this is example usage:

# Markdown is Prose, but we still want it in the stats
*.md gengo-detectable

# JavaScript is Programming, but we want to exclude it
*.js -gengo-detectable

# The contents of dist/ are not generated, so they will be detectable
dist/* -gengo-generated
# OR
# Even though dist/* is generated, it should still be included in the stats
dist/* gengo-detectable

By default any file that is not detectable is excluded from the Summary.

---

Again, the Git file source is highly opinionated and tries to behave a lot like github-linguist. The usage with the Directory file source is still undecided, and right now it doesn't implement any overrides. Or we could write our own file source (#1303 (reply in thread)) if we want very specific behavior that wouldn't be suitable for the gengo crate.

But, however we implement it, IMO the best way would be to inform gengo what files are and aren't detectable, and then make use of either the Analysis (detailed results) or Summary (simplified results).

---

tl;dr Yes, gengo would be the one handling included/excluded types, and we'd just pass configuration to it somehow.

[o2sh]

Are the pending changes taken into count for the language analysis?

[spenserblack]

#1303 (comment)

[spenserblack]

Alright, after a long time procrastinating, I implemented the last of onefetch's languages (besides combining Bash, Zsh, and Sh into Shell).

One thing to note is that the MSRV has been bumped.

[spenserblack]

Bumping gix resulting in compiler errors (new usage). Just a moment.

[spenserblack]

After thinking about this a bit, I think an issue with tokei and other line-counting tools is that they specialize in counting lines, and, while language ID is a priority, not the top priority. Which leads to e.g. tokei having poor out-of-the-box support for Verilog and V (their *.v extensions were already taken). So line-counting tools are not great for properly identifying a language.

So I'd like to add line counts back at some point, possibly by using gengo as a dependency of some crate and doing something vaguely like

match Language {
    JupyterNotebook => my_super_special_jupyter_notebook_counter()
    _ => generic_counter(Language) // would use community-provided data similar to tokei's languages.json
}

And on that topic, I'd recommend looking into scc. It's a Go project similar to tokei, but also includes some really fun stats like "Estimated Cost to Develop".