Add julia parser #2654
Add julia parser #2654
Conversation
|
I would like you to add "Julia" to docs/news.rst. |
|
Could you add "julia.c" to win32/ctags_vs2013.vcxproj and win32/ctags_vs2013.vcxproj.filters. |
|
Thank you for your contribution. I must say this first. |
| { true, 's', "struct", "Structures" }, | ||
| { true, 't', "type", "Types" }, | ||
| { true, 'x', "unknown", "Imported name"} | ||
| }; |
There was a problem hiding this comment.
In u-ctags, we use "reference tag" to tag for X in "import X" because
"import X" doesn't define X. X is defined somewhere. "import X" refers it.
See #2428,
https://docs.ctags.io/en/latest/man/ctags-lang-python.7.html, and parsrs/python.c.
As far as I can remember, I don't write any document the way to make reference tags. So whether I can ask you to use a reference tag for extracting X in "import X". I can work on this topic instead of you. Tell me if I should do (after merging your parser). In such a case, what I'm afraid of is that the impact of my change to Geany.
You are working on Julia's support in Geany. Introducing reference tags may break your work partially.
To avoid it, we have to cooperate.
There was a problem hiding this comment.
I don't understand how reference tags work. Can you explain a bit? I found it in python.c and thought it was appropriate for julia too. Just for you to know, in Julia, imports work like in python from <module> import * so I don't know if looking for definition in another file would work...
There was a problem hiding this comment.
The description of reference tags for users:
By default, ctags records only definition tags.
If you also want ctags to record reference tags, specify --extras=+r.
A reference tag has a kind like function, variable, etc. This is the same as definition tags.
In addition, a tag has roles. A role is a subcategory of a kind. The roles are not shown by
default. specify --fileds=+r.
x = 1
y++
def z
In this example
x ... kind:variable roles:assigned
y ... kind:variable roles:assigned,incremented
z ... kind:varaible roles:def
def in roles means z is tagged as a definition tag.
x and y have no def' in roles` field. Therefore, they are tagged as reference tags.
What kind of roles other than def is defined can be listed with --list-roles option.
$ ./ctags --list-roles=Python
#KIND(L/N) NAME ENABLED DESCRIPTION
i/module imported on imported modules
i/module indirectlyImported on module imported in alternative name
i/module namespace on namespace from where classes/variables/functions are imported
x/unknown imported on imported from the other module
x/unknown indirectlyImported on classes/variables/functions/modules imported in alternative name
$ ./ctags --list-roles=Python.'{module}'
#KIND(L/N) NAME ENABLED DESCRIPTION
i/module imported on imported modules
i/module indirectlyImported on module imported in alternative name
i/module namespace on namespace from where classes/variables/functions are imported
CURRENTLY, Reference tags are not popular.
However, in a limited area, I'm pushing introducing reference tags. See #2428.
The description of reference tags for parser developers.
Follow the steps to introduce a reference tag to your parser.
- define a role in a kind like:
typedef enum {
...
K_MODULE,
...
COUNT_KIND
} pythonKind;
typedef enum {
PYTHON_MODULE_IMPORTED,
PYTHON_MODULE_NAMESPACE,
PYTHON_MODULE_INDIRECTLY_IMPORTED,
} pythonModuleRole;
static roleDefinition PythonModuleRoles [] = {
{ true, "imported",
"imported modules" },
{ true, "namespace",
"namespace from where classes/variables/functions are imported" },
{ true, "indirectlyImported",
"module imported in alternative name" },
};
/* if a kind is never used for making a definition tag,
set `referenceOnly` to true. */
static kindDefinition PythonKinds[COUNT_KIND] = {
...
{true, 'i', "module", "modules",
.referenceOnly = true, ATTACH_ROLES(PythonModuleRoles)},
...
};
- make a referenec tag like
tagEntryInfo e;
int kindIndex = K_MODULE;
int roleIndex = PYTHON_MODULE_NAMESPACE;
initRefTagEntry (&e, "tagNameInCString",
kindIndex, roleIndex);
...
markTagExtraBit (&e, XTAG_REFERENCE_TAGS);
return makeTagEntry (&e);
or
makeSimpleRefTag (tagNameinVString, K_MODULE, PYTHON_MODULE_NAMESPACE);
|
Hi, thank you, I added julia.c to the 3 files you mentioned and I changed Tabs to spaces and remove trailing spaces, I hope the code is more readable like this. |
|
Is there a library in ctag to help deal with unicode? It's because julia makes a big use of unicode identifiers and operators, so it would be good if they are well parsed (currently unicode operators are parsed as identifiers). |
|
|
The C compiler on Windows says somthing: |
No, there isn't. However, you may be able to study some from parsers/json.c. |
|
BTW, I found an interesting document: https://docs.ctags.io/en/latest/contributions.html# |
|
A test case is failed. Your parser extracted |
|
I tried the parser. Run Some vStrings are not deleted. |
Codecov Report
@@ Coverage Diff @@
## master #2654 +/- ##
==========================================
+ Coverage 86.99% 87.10% +0.10%
==========================================
Files 185 186 +1
Lines 39465 39947 +482
==========================================
+ Hits 34334 34796 +462
- Misses 5131 5151 +20
Continue to review full report at Codecov.
|
|
@getzze, though I described the way to implement reference tags, I think we can implement them in Julia later. |
|
@getzze, could you introduce me larger git repositries where source files written in Julia are stored? (if you know) I would like to add the repositories to https://github.com/universal-ctags/codebase. |
|
Great, thanks! Here is the list of most of the known repos:
Julia standard libraries:
https://github.com/JuliaLang/julia/tree/master/stdlib
JuliaLang repo contains other standard libraries that are kept apart from the main repo (but are still standard like Pkg.jl, the Julia package manager). The packages are the repos that end with '.jl':
https://github.com/JuliaLang
General registry with all registered packages. For each package, there is a file Package.toml with entry `repo` that points to an url with the source files:
https://github.com/JuliaRegistries/General
…On 6 October 2020 17:15:25 GMT+01:00, Masatake YAMATO ***@***.***> wrote:
@getzze, could you introduce me larger git repositries where source
files written in Julia are stored? (if you know)
I would like use it for testing your parser.
I would like to add the repositories to
https://github.com/universal-ctags/codebase.
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#2654 (comment)
|
|
Thank you. |
|
@getzze, though I described the way to implement reference tags, I think we can implement them in Julia later.
I think we should focus on merging the current implementation.
Yes it's probably best. Julia heavily uses multiple dispatch (like function overloading), therefore each function can be defined several times, each of such definition is called a method (common functions like `convert` have hundreds of definitions!). Knowing what is the corresponding method definition for each function call would depend on the argument types, that are often only known at run time...
In short, it will be a mess.
|
parsers/julia.c
Outdated
| /* Resets the scope string to the old length */ | ||
| static void resetScope (vString *scope, size_t old_len) | ||
| { | ||
| scope->length = old_len; |
There was a problem hiding this comment.
You can use vStringTruncate() here.
|
Can I take over this PR? I would like to make more minor changes. |
Great catch. I am not very satisfied of how short functions are parsed, because it is very prone to bugs, if you have a better implementation it would be great. The problem is to parse an Right now, several cases are missed:
@masatake edited the style of indentation. |
|
Do you have a plan to introduce more changes? Anyway rebasing on the master branch is welcome. So reviewing becomes easier. |
|
Sorry, I thought pulling your changes before committing would not disturb. PS: the reference tags are working fine in Geany! |
I see.
Interesting. Could you tell more? I would like to know how Geany utilizes the reference tags. |
No, this doesn't work, especially in Julia where you have several files X, for each version of the package. |
|
Oh, I see. Thank you. |
I don't understand the problem well because I don't know Julia. Do I have to study the syntax of Julia? In that case, it will take much more time to merge this parser. I need examples that include input files and expected tags and actual tags. |
|
I think you don't have to study Julia syntax, I was just trying to explain in case there was an example (that you coded for ctags) in another language where a function is defined without keyword. So that I could use it as an example for the julia parser. I pushed a more extensive testing unit file like you asked, it should help improve coverage too. Do you need help from me to finish reviewing the parser? |
|
o.k.
Could you show examples of the above sentences in Julia language? ... I'm sorry. You already showed examples in the comment #2654 (comment). I missed the comment. I will review the changes again. |
|
In #2654 (comment) you showed some code examples. |
|
I added the expected tags in #2654 (comment). |
|
roundtrip test harness reports a failure about alpha. |
The original bug was found when developing Juila parser in u-ctags
repo[1]. When searching a tag name "α", readtags reported nothing
though an entry for "α" was in the tags file used for testing
the Julia parser.
"α" is {206, 177} as a byte sequence.
However, when comparing bytes in strings, libreadtags used (singed)
char type. In the case "α" was {-50, -79}. Using {-50, -79} for
comparison, the binary search didn't work as we expected.
The test for this change will be done in u-ctag side.
[1] universal-ctags/ctags#2654
Signed-off-by: Masatake YAMATO <yamato@redhat.com>
|
A bug of libreadtags causes the failures of testing on the CI environments. So please ignore the failures for a while. |
|
Thank you for updating the comment. I don't know Julia. So I always need the pair (input and expected output) to understand what you write. Obviously, we can't deal with eval. So let's forget it. Could you rebase the changes of this pull request onto the latest HEAD of the master branch, and push --force here again? |
|
Force updated. |
Signed-off-by: Masatake YAMATO <yamato@redhat.com>
|
I would like to review code tagging language objects related to "import". |
|
Thank you very much. |
Tested with Geany (geany/geany#2584)