-
Notifications
You must be signed in to change notification settings - Fork 624
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PuppetManifest: ctags command never finishes on long puppet file #1912
Comments
Attached to the process with lldb and this is the stack trace I got repeatedly couple of times
|
To further debug, I think I need debug-info. If there is a build flag that will generate the debug info, I would appreciate any help to set that up. |
C source level debugger doesn't help for optimizing the mtable parser. |
Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 142.32s user 21.10s system 99% cpu 2:43.67 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 73.08s user 17.89s system 99% cpu 1:31.16 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 73.08s user 17.89s system 99% cpu 1:31.16 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 49.81s user 15.95s system 99% cpu 1:05.91 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 49.81s user 15.95s system 99% cpu 1:05.91 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 46.45s user 15.58s system 99% cpu 1:02.12 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Finally, I myself found |
…erals Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 46.45s user 15.58s system 99% cpu 1:02.12 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 4.48s user 1.01s system 99% cpu 5.498 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
About 30 times faster. |
Please, try #1913. |
I have checked out masatake/puppetManifest-optimization, cleaned and re-compiled. I still see that ctags is stuck at 100% cpu over several minutes.
|
I run example.pp. The new code is about 30 times faster than before. However, I don't say it completes in 10 minutes. If it took a day on your pc, It will take an hour with the new code. What happens if you make example.pp smaller? |
OK Fair enough. I think your changes addressing this issue are extremely useful. Thanks for your help here, again. I ran the following experiment and the 40K line input did complete. The files are named
Having said that, I see a problematic tendency here. As a user, I must say this is surprising. Again, not knowing about the implementation, sorry if I make wrong assumptions here. If I recall correctly, other source code parser's (like compilers) runtime does not increase quadratically with increasing source code size. Sorry if I am not doing a just comparison here. |
Could you upload |
Reproduced. I expected O(n). I cannot find a good explanation for O(n^2). |
Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 142.32s user 21.10s system 99% cpu 2:43.67 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 73.08s user 17.89s system 99% cpu 1:31.16 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 73.08s user 17.89s system 99% cpu 1:31.16 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 49.81s user 15.95s system 99% cpu 1:05.91 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 49.81s user 15.95s system 99% cpu 1:05.91 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 46.45s user 15.58s system 99% cpu 1:02.12 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
…erals Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 46.45s user 15.58s system 99% cpu 1:02.12 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 4.48s user 1.01s system 99% cpu 5.498 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 4.48s user 1.01s system 99% cpu 5.498 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 3.46s user 0.84s system 99% cpu 4.312 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Sorry for the delay. |
Ok Great. This is good news. In my experience, setting up valgrind was time consuming and tedious. But once that is done, you generate a call graph. There you can see which functions are called N times and which ones N^2 times. That information helps with the diagnosis significantly. Sorry If I am stating the obvious here. |
What do you say we close this issue with the improvements you have done and merge them? Because technically the 40K line file's processing does not take indefinitely anymore. Then we can create another issue for the O(N^2) behavior. |
Conceptually, the puppetManifestParser is not written in C. I should know the reason of O(n^2). However, I must work on another item about Go parser. |
O.k. I will merge the pull request associated with this issue. Then I will clsoe this. |
Thank you very much. I will create a new issue. Even if it will not be fixed soon, it is important to keep track of it. |
I have a ~40K line long puppet file. ctags never finishes processing it. (I left it at lest 3-4 hours). Ctags is stuck at 100% cpu.
The example.pp file is attached.
example.pp.zip
The file looks like this
A half certain observation: If I remove all the params from all the declarations (param_1 to param_8) ctags does indeed complete. ( The owner of the issue needs to reproduce this. I think I am correct but It is worth to retry this one more time)
The ctags process looks like this in my mac's top -o cpu` output
example.pp is indeed valid puppet file
The name of the parser: puppetManifest
The command line you used to run ctags:
The content of input file:
see above
The tags output you are not satisfied with:
no tags generated. ctags stuck
The tags output you expect:
Do not know at this point
The version of ctags:
How do you get ctags binary:
Built it from source.
The text was updated successfully, but these errors were encountered: