No language being detected for NetBSD repo

[jamesqo]

This mirror repo of the NetBSD source code:

https://github.com/IIJ-NetBSD/netbsd-src

doesn't appear to have a language detected, even though it's full of plenty of examples of C code.

Also, may/may not be related, but this repo doesn't have one detected either, while it contains a bunch of C# files.

[pchaigno]

Also, may/may not be related, but this repo doesn't have one detected either, while it contains a bunch of C# files.

The C# files in samples/ are ignored in statistics by Linguist because the samples folder is considered documentation. You can change that with Linguist overrides, in a .gitattributes file:

samples/* linguist-documentation=false

This mirror repo of the NetBSD source code:
https://github.com/IIJ-NetBSD/netbsd-src
doesn't appear to have a language detected, even though it's full of plenty of examples of C code.

I suspect that this is happening because of the huge size of the repository (6.2G with .git). I made a copy of the repository, removed the external folder (2.1G) and re-initialized .git (git init). Linguist works fine on this copy (900M with .git).

[arfon]

I suspect that this is happening because of the huge size of the repository (6.2G with .git). I made a copy of the repository, removed the external folder (2.1G) and re-initialized .git (git init). Linguist works fine on this copy (900M with .git).

Yeah, I tried to manually run the language detection script from our admin tools and I'm afraid it doesn't complete.

[pchaigno]

Even locally it fails to complete (which would indicate that the problem comes from Linguist itself...).

[arfon]

Even locally it fails to complete (which would indicate that the problem comes from Linguist itself...).

I suspect the language calculation job is timing out somewhere in our infrastructure either because of memory usage and/or the time the job takes to run.

I don't see us prioritizing this anytime soon sorry because it feels like an edge case for very large repositories. I'm therefore going to close this.

[pchaigno]

Linguist currently returns no language counts - even when run locally - because of this line, which limits the number of blobs in the repository to 100,000.

If I increase the number of blobs to 200,000 (there are around 196k blobs in that repository), Linguist takes around 3 minutes to execute on netbsd-src on my computer (running on a single 2.4GHz core).
This is only because it's the initial run. Since the computation of statistics is incremental, subsequent computations would be much faster.

[arfon]

Linguist currently returns no language counts - even when run locally - because of this line, which limits the number of blobs in the repository to 100,000.

Good find @pchaigno. I'll see what the systems/infrastructure team think about bumping the file limit for production.

[sanajitjana]

The same problem - language was not detected for this repo https://github.com/sanajitjana/react-map-method

[lildude]

@sanajitjana It looks like the analysis job didn't run or complete successfully when you pushed. Try pushing another change and that should kick off the job.