Add Euphoria language

[ghaberek]

Description

Add support for the Euphoria language (website, github). We have been gradually migrating development of Euphoria onto GitHub but the frequent mis-classification by linguist makes it look like we're using a handful of other languages (namely E, Eiffel, and Elixir) due to extension collisions. Euphoria primarily uses .e for include files and .ex for programs. These changes should correct that. I've done some diligence in correcting new false-positives by adding a sample for Elixir and several patterns to the heuristics, although admittedly I am not nearly as familiar with those other languages. I did test linguist against several larger Eiffel and Elixir repositories.

Checklist:

• I am adding a new language.
  • The extension of the new language is used in hundreds of repositories on github.com.
    • Search results for each extension:
      • .e - https://github.com/search?q=procedure+extension%3Ae&type=Code&ref=advsearch&l=&l=
      • .ex - https://github.com/search?q=function+-def+-defmodule+-import+-interface+-nil+extension%3Aex&type=Code&ref=advsearch&l=&l=
  • I have included a real-world usage sample for all extensions added in this PR:
    • Sample source(s):
      • Eiffel
        • app_embedded_web_execution.e
        • app_embedded_web_service.e
        • desktop_app.e
        • main_window.e
      • Elixir
        • atom.ex
        • float.ex
        • function.ex
        • integer.ex
        • regex.ex
      • Euphoria
        • allsorts.ex
        • animal.ex
        • csort.ex
        • debug.e
        • dll.e
        • hash.ex
        • math.e
        • mydata.ex
        • regex.e
        • types.e
    • Sample license(s):
      • Eiffel: EFL
      • Elixir: Apache License 2.0
      • Euphoria: MIT license
  • I have included a syntax highlighting grammar: https://github-lightshow.herokuapp.com/?utf8=%E2%9C%93&scope=from-url&grammar_format=auto&grammar_url=https%3A%2F%2Fgit.luolix.top%2FOpenEuphoria%2Fvscode-euphoria%2Fblob%2Fmaster%2Fsyntaxes%2Feuphoria.tmLanguage.json&grammar_text=&code_source=from-url&code_url=https%3A%2F%2Fgit.luolix.top%2FOpenEuphoria%2Feuphoria%2Fblob%2Fmaster%2Fsource%2Fmain.e&code=
  • I have updated the heuristics to distinguish my language from others using the same extension.

[ghaberek]

Apparently I didn't properly commit the git submodule for vscode-euphoria, probably because I synced my fork at some point and left that out. I've committed the submodule now, and that corrected the failing Ruby tests.

I'm still not sure how to correct the cross-validation test. I'm not even sure what's wrong. Running bundle exec script/cross-validation --test isn't covered in the contributing doc but bundle exec rake test completes successfully on my system.

[lildude]

This is looking great. Thanks. 🙇 One nit about the extensions inline and an explanation for the failing test:

The classifier test uses just the samples in Linguist to test the classifier in isolation. It is failing because the classifier is detecting the Elixir/function.ex file as Eurphoria (it's buried here):

Elixir/function.ex BAD (Euphoria)

This will be happening because the content of that sample matches more attributes of Euphoria based on the samples provided than Elixir, which isn't really surprising because Elixir has only one other file and it's teeny weeny and really not representative of the language.

The solution is to add another Elixir .ex file that is a good representation of the language.

This wasn't an issue before now because Elixir was was added to Linguist many many years before we started curating samples so one wasn't added and it was the only language using .ex until now 😁 .

And thanks the for the pointer that the docs are missing mention of running this test.

[ghaberek]

I added several more samples for Elixir and that seems to have pleased the cross-validation script. 👍

[ghaberek]

Oops, didn't realize I need to mark that resolved. Also pulled in the latest change from upstream. 👍

[ghaberek]

Just to clarify, is there anything else needed from my end to get this accepted?

[lildude]

Not at the moment thanks. @Alhadis may have some suggestions around the regexes when he gets a chance to have a look, but nothing else is needed at the mo.

[lildude]

@ghaberek can you please address the conflicts 🙇

@Alhadis when you've got a mo, do you mind taking a look at these regexes 🙇

[lildude]

Nudge

[Alhadis]

@Alhadis when you've got a mo, do you mind taking a look at these regexes 🙇

Crap. 😞 Sorry. I'll have a look now.

[lildude]

Nudge @ghaberek 😉

[lildude]

Most of these samples are massive because it appears (from my naïve perspective) that they contain a lot of comments and a fair amount of duplication.

Please swap these for smaller samples with fewer/smaller comments.

[ghaberek]

Please swap these for smaller samples with fewer/smaller comments.

All set. Unless those are still too big? A lot other files in the samples/ directory are well over 1k lines.

[lildude]

Much better samples, thanks. 🙇

[lildude]

Much better samples, thanks. 🙇

Oh, maybe not.

Tests are failing with these samples.

[ghaberek]

Tests are failing with these samples.

TestHeuristics#test_ex_by_heuristics [/home/runner/work/linguist/linguist/test/test_heuristics.rb:23]:
no fixtures for Euphoria *.ex

So I just need to provide some .ex samples in the samples/Euphoria folder?

[lildude]

So I just need to provide some .ex samples in the samples/Euphoria folder?

I don't think so.

Update: Whoops, yes... for that error.

For the current failures:

I think you need to improve your heuristic as it's not detecting one of the files you're supplying correctly so leaving it to chance with the classifier.

If we're going to add samples, it would be for Eiffel as there are only 3 very old samples, but I think improving the heuristic would be a better approach for now.

[ghaberek]

I don't think so.

Update: Whoops, yes... for that error.

I figured that so added more Euphoria samples with .ex yesterday.

I think you need to improve your heuristic as it's not detecting one of the files you're supplying correctly so leaving it to chance with the classifier.

The original Eiffel heuristic I added was only looking for left-anchored keywords, but since any Eiffel keyword is also a valid identifier in Euphoria, things like from = 2 in Euphoria would match Eiffel's keyword pattern as ^\s*(?:from)\s. I've added two new patterns to the Eiffel heuristic to match variable and function declaration syntax.

If we're going to add samples, it would be for Eiffel as there are only 3 very old samples, but I think improving the heuristic would be a better approach for now.

I've added a few more Eiffel samples.

[ghaberek]

Thanks for accepting this! It's going to be a big help to our community. It looks like syntax highlighting is working now on Euphoria files in my projects, but I'm not seeing the "Languages" statistic update on anything yet. What does it take to make that happen?

[Nixinova]

Looks fine to me

[ghaberek]

Strange. I went back to the OpenEuphoria page and it was correct there as you showed but not on the Euphoria project itself.

I just took this screen shot. But then I did a Shift+F5 and it updated to what you show. False alarm I guess. Thanks again!

[lildude]

You need to push a change to the repo that will cause a re-analysis. Changes to .gitattributes, the only file changed in that repo since that new release was deployed, does not cause a re-analysis.