Add STAR language and .star file extension for Starlark

[mcuadros]

Description

This PR's adds .star as an extension to the Starlark language. The .star extension is the default extension for Starlark scripts not related to Bazel, this is growing every day since many projects are embedding the language in their own projects.

This extension is used in all official interpreters in the examples and as well in the testdata.

Also is the official extension recommended by many opensource projects:

• https://docs.drone.io/pipeline/scripting/starlark/#context
• https://github.com/qri-io/qri/tree/master/startf/testdata
• https://github.com/mcuadros/ascode/tree/master/_examples

This was discarded initially at #4759 (comment)

The amount of .star files currently is 30k, since the PR instrudicing the language the usage has growth a 50%.

Checklist:

• I am associating a language with a new file extension.
  • The new extension is used in hundreds of repositories on github.com
    • Search results for each extension:
      • https://github.com/search?utf8=%E2%9C%93&type=Code&ref=searchresults&q=extension%3Astar+NOT+nothack
  • I have included a real-world usage sample for all extensions added in this PR:
    • Sample source(s):
      • https://github.com/dockhippie/nodejs/blob/a04a35ef94418d559b434aa4834b9e71960fd42f/.drone.star
      • https://github.com/gn1484/grafana/blob/a0b707ebe0807f286fabb5a6462fc205459803d1/scripts/drone/pipelines/release.star
    • Sample license(s): MIT & AGPL3 respectively
  • I have included a change to the heuristics to distinguish my language from others using the same extension.

@lildude has also added support for STAR as it also has the .star extension which is very popular on GitHub:

• I am adding a new language.
  • The extension of the new language is used in hundreds of repositories on github.com.
    • Search results for each extension:
      • https://github.com/search?q=extension%3Astar+loop_
  • I have included a real-world usage sample for all extensions added in this PR:
    • Sample source(s):
      • https://github.com/fullerjamesr/mvf/blob/644336514a32efbca16105cefe3cd87672867be0/relion/Schedules/mvf_basic/schedule.star
      • https://github.com/fullerjamesr/mvf/blob/644336514a32efbca16105cefe3cd87672867be0/relion/Schedules/mvf_basic/schedule_pipeline.star
      • https://github.com/Vilax/xmipp-lite/blob/00602dd4f3795cb8fea799de8f8681c50282f979/src/xmipp/resources/test/metadata/symop.star
    • Sample license(s): AGPL 3.0, AGPL 3.0 and GPLv3 respectively
  • I have included a syntax highlighting grammar: No
  • I have updated the heuristics to distinguish my language from others using the same extension.

[pchaigno]

I don't think this pull request addresses the concerns expressed in #4759. The issue was not with the in-the-wild usage, but with the presence of many other .star files. All these files will be recognized as Starlark files with the pull request as is.

[mcuadros]

Sorry, but if I am not mistaken, in linguist are many extension collisions, that are resolved with the classifier. For example `.md`. So all this files are not going to be identified as Starlark since a sample is provided for the classifier. I am wrong? Usually this can be a problem because this consume more resources than just check the extension. But given the number of .star is slow, I don't believe this is going to be a problem.

…

On Sun, Apr 19, 2020, 9:47 PM Paul Chaignon ***@***.***> wrote: I don't think this pull request addresses the concerns expressed in #4759 <#4759>. The issue was not with the in-the-wild usage, but with the presence of many other .star files. All these files will be recognized as Starlark files with the pull request as is. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4840 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMAB6XTV43Z3ETYLJCPWRDRNNIOZANCNFSM4MF2JEBA> .

[pchaigno]

Sorry, but if I am not mistaken, in linguist are many extension collisions,
that are resolved with the classifier. For example .md.

So all this files are not going to be identified as Starlark since a sample
is provided for the classifier. I am wrong?

That is correct, but requires us to also have support in Linguist for the conflicting languages. In the case of .md, Linguist knows both GCC Machine Description and Markdown and can use the classifier (and other, more precise strategies) to distinguish between the two. So for .star, we'll need to identity the conflicting languages and add them to Linguist if they meet usage requirements.

[mcuadros]

Well but if the other languages are not identified will remain the same, but the Starlark files will be identified. Even if doesn't work like this, the amount files that aren't Starlark it's minimal. And Starlark is growing everyday. Please reconsider it.

…

On Sun, Apr 19, 2020, 10:52 PM Paul Chaignon ***@***.***> wrote: Sorry, but if I am not mistaken, in linguist are many extension collisions, that are resolved with the classifier. For example .md. So all this files are not going to be identified as Starlark since a sample is provided for the classifier. I am wrong? That is correct, but requires us to also have support in Linguist for the conflicting languages. In the case of .md, Linguist knows both GCC Machine Description and Markdown and can use the classifier (and other, more precise strategies) to distinguish between the two. So for .star, we'll need to identity the conflicting languages and add them to Linguist if they meet usage requirements. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4840 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMAB6SJLQKC67N5CDGSAQ3RNNQCRANCNFSM4MF2JEBA> .

[stale]

This pull request has been automatically marked as stale because it has not had recent activity, and will be closed if no further activity occurs. If this pull request was overlooked, forgotten, or should remain open for any other reason, please reply here to call attention to it and remove the stale status. Thank you for your contributions.

[stale]

This pull request has been automatically closed because it has not had activity in a long time. Please feel free to reopen it or create a new issue.

[fkorotkov]

@pchaigno I want to bump this issue. What will require for us to "identity the conflicting languages"?

I made a search query to look for recently indexed *.star files, checked a few first pages and it seems majority of cases are Startlark cases. The rest is some data files or files without a particular pattern.

[mahmoudimus]

bump! seconding @fkorotkov. /cc @lildude @pchaigno - any way we can get this looked at?

[lildude]

bump! seconding @fkorotkov. /cc @lildude @pchaigno - any way we can get this looked at?

Things are still in the same place as it was... you still need to identify the other users of the .star extension. From a very quick look, there's one particular use of .star files easily identifiable that is pretty active with nearly 8000 files and each file appears to occur only once per repo making it a clear example that qualifies for inclusion else it would be incorrectly identified. I have no idea what language it's written in, but it appears to be related to machine learning from the few READMEs I've looked at so it will be noticed very quickly if it's misidentified.

You don't have to identify every user of the .star extension, but an attempt needs to be made to at least identify the most popular (probably just the one I've found), and then add them (it) as part of this PR (I've re-opened it).

[fkorotkov]

Just for a reference we at Cirrus CI started using Starlark along side YAML for allowing to programmatically configure CI builds. So there will be more .cirrus.star and lib.star files on GitHub in the near future. 😅

We also created a plugin for IntelliJ platform to support code assistance for .star files.

[rohansingh]

@lildude Just so I understand, if we go through and identify all the conflicting languages that meet usage requirements, do they need to be added as new languages in this same PR?

[lildude]

@rohansingh I think you may have missed the last sentence in my last response:

You don't have to identify every user of the .star extension, but an attempt needs to be made to at least identify the most popular (probably just the one I've found), and then add them (it) as part of this PR (I've re-opened it).

[rohansingh]

Sorry, reading comprehension failure. Anyway, sounds good, I'll try to get to this today.

update: Unfortunately I don't see myself getting around to this after all. Just using .gitattributes as a workaround for now.

[lildude]

Marking this as still needing changes

[lildude]

You don't have to identify every user of the .star extension, but an attempt needs to be made to at least identify the most popular (probably just the one I've found), and then add them (it) as part of this PR (I've re-opened it).

I've done this. I've identified the other most commonly used language is Self-defining Text Archive and Retrieval commonly referred to as STAR.

I've pushed changes to your repo to add support to this PR.

[Alhadis]

That pro-dna.nmr.star sample is huge (8,613 lines / 145 KBs). Sure we can't trim it down a bit? I noticed that a lot of sections repeat the same data.

[lildude]

That pro-dna.nmr.star sample is huge (8,613 lines / 145 KBs). Sure we can't trim it down a bit? I noticed that a lot of sections repeat the same data.

🤦 And I point this out myself so often 😆 I'll get the ✂️ out.

[lildude]

@Alhadis replaced it with a smaller sample.