Q: What’s the policy on duplicate extensions?

[wooorm]

Hi!

Macro Q: I was wondering what the policy was on duplicate extensions. I image there to be duplicates for niche languages, but if there’s a clear super popular use of an extension, should others be removed?

Micro Q: Particularly I see that .tsx is included in the XML language, and .md in some GCC Lisp flavor.

---

I think extensions affect how markdown fenced code works here on GH.
But regardless, extensions here will also affect other tools using the provided data.
From what I can tell, I think the grammars here are loaded into the GitHub app by index, which is case sensitive, which means that .tsx is used for TS (as it comes before XML), whereas GCC is used for .md, which comes before markdown.
My assumption here is that most people would want ```md to highlight as markdown, not Lisp.

Note: the highlighting here on GH does not match how it renders on GH:

```md
let a = 123; /* Lisp? */

# markdown?
```

```tsx
let a = 123; /* TS? */

<!--xml?-->
```

Note: See that Lisp wins over markdown, and TS over XML.

let a = 123; /* Lisp? */

# markdown?

let a = 123; /* TS? */

<!--xml?-->

Proposal:
a) I think some policy might be useful
b) I can make a PR to remove .tsx from XML and .md from GCC Lisp, there might be others!

[lildude]

I was wondering what the policy was on duplicate extensions.

Linguist's policy is loosely detailed in the CONTRIBUTING.md file, but this is specific to the analysis of individual files. In short, we accept duplicates and rely on samples to train the classifier and heuristics to differentiate the different languages based on the content of the files that share a common extension.

This however has no influence on what you are reporting as the rendering of codeblocks is done entirely by the markup library and it'll only highlight the blocks with the language YOU tell it to use.

I think extensions affect how markdown fenced code works here on GH.

They do, however they're not the only thing. The language name and aliases defined in the languages.yml file are also considered by the markup library that does the actual rendering for this precise reason: to remove ambiguity.

And this is where the crux of your issue is...

My assumption here is that most people would want ```md to highlight as markdown, not Lisp.

You are indeed making an assumption, and someone else may be making the inverse assumption. The markup library has no way of knowing if your assumption or their assumption is the intended result, so in the cases of ambiguity, it goes for the first in the list, as ordered by the languages.yml file (alphabetical by language name). You're also assuming most people use the extension for the codeblock which isn't the case. Most users use the language name or one of the aliases as the extension can be ambiguous, as you've found.

So in order to get the correct highlighting in cases of ambiguousity, be more precise: use the language name, or an alias, that YOU want.

I want this block to be highlighted as markdown so I use: ```markdown```

let a = 123; /* Lisp? */

# markdown?
**bold**

I want this block to be highlighted as XML so I use: ```xml```

let a = 123; /* TS? */

<!--xml?-->
<xml>foo</xml>

I want this block to be highlighted as TSX so I use: ```tsx```

let a = 123; /* TS? */

import * as React from 'react'

I want this block to be highlighted as ASP.NET but it's a pain to write that out so I use the alias: ```aspx```

<%@ Register Src="~/Account/OpenAuthProviders.ascx" TagPrefix="uc" TagName="OpenAuthProviders" %>

I really don't know what to use for this block so I don't specify a language:

let a = 123; /* Lisp? */

# markdown?
**bold**

<!--xml?-->
<xml>foo</xml>

import * as React from 'react'

<%@ Register Src="~/Account/OpenAuthProviders.ascx" TagPrefix="uc" TagName="OpenAuthProviders" %>

So in short, YOU are being ambiguous with your request to markup when it comes to highlighting your codeblock and in turn have hightlighted a short-coming with using extensions to indicate the language of a codeblock that only comes up when using extensions shared by languages. The only solution is to be more explicit about what YOU want; neither Linguist nor markup can guess this.

Proposal:
a) I think some policy might be useful

We've already got one. We accept duplicates and use samples to train the classifier and heuristics to make the language selection more accurate when analysing files. When it comes to codeblocks, this has nothing to do with Linguist and such a policy would need to be documented in the markup docs, though it really comes down to "YOU are telling markup what language to use, so be more precise about what you want it to do; it can't guess".

b) I can make a PR to remove .tsx from XML and .md from GCC Lisp, there might be others!

Don't. This will be rejected.

[wooorm]

You're also assuming most people use the extension for the codeblock which isn't the case. Most users use the language name or one of the aliases as the extension can be ambiguous, as you've found.

To illustrate my assumption with numbers, for JavaScript, which supports the full name and the extension:

• https://github.com/search?q=path%3A.md+%60%60%60js&type=code js has 3.2m hits
• https://github.com/search?q=path%3A.md+%60%60%60javascript&type=code javascript has 1.4m hits

…folks seem to prefer extensions.

For markdown, there are half as many people using ```md as ```markdown.

• https://github.com/search?q=path%3A.md+%60%60%60markdown&type=code markdown has 99k hits
• https://github.com/search?q=path%3A.md%20%60%60%60md&type=code md has 43k hits

That’s a lot of people intending markdown, but getting Lisp. 30k repos.

Note that even GitHub’s own syntax highlighting for markdown, included here in linguist too, supports ```md for markdown.

Note: there are probably some false-positives in the shorter searches (```js and ```md), as they match different things. Still, it’s a lot of hits that from glancing at them, mostly matches.

I’d expect that a large share of the folks using ```markdown tried the extension first, found out that it didn’t work, and then changed it.

YOU

I am not sure why you’re painting this request as some fringe weird idea? The numbers above are quite large. Even if it was a couple thousand of folks making this “mistake”, at some point it’s perhaps better to not tell them they’re wrong?

[lildude]

That’s a lot of people intending markdown, but getting Lisp. 30k repos.

Note that even GitHub’s own syntax highlighting for markdown, included here in linguist too, supports ```md for markdown.

Note: there are probably some false-positives in the shorter searches (```js and ```md), as they match different things. Still, it’s a lot of hits that from glancing at them, mostly matches.

I’d expect that a large share of the folks using ```markdown tried the extension first, found out that it didn’t work, and then changed it.

I agree, though these two aren't quite the same thing. Markdown shares an extension whilst Javascript doesn't, and it's much easier to type js than javascript... and as I type this I've just realised something I missed before: we don't have "md" as an alias for Markdown. 🤦 Adding it as an alias will solve most, if not all, of these ```md => Lisp cases.

I am not sure why you’re painting this request as some fringe weird idea? The numbers above are quite large. Even if it was a couple thousand of folks making this “mistake”, at some point it’s perhaps better to not tell them they’re wrong?

I apologise for my wording. I didn't mean to transfer blame or indicate this was a fringe weird idea. It was intended as emphasis from a third-person perspective where "you" is the reader/user/author writing the codeblock, or reading my response later, that is in control of dealing with the ambiguity in such cases. No analysis of the content within codeblocks takes place so there is no way for markup to know the correct answer without being told.

I've opened #6338 to add md as an alias for Markdown. This won't go live until the next release which won't be for a whilst as I've just made a release.

Thanks for bringing this to my attention and making me realise this could be resolved from Linguist 🙇

[wooorm]

No worries! Coool :) Awesome to see this get fixed!