Ignoring git ignored folders per project

[ahmedelgabri]

I'm trying to make gutentags run only on files that are tracked by source control git in my case. But I can't get it right, so I need to some pointers.

here is my configuration

set tags=./.tags,.tags;

let g:gutentags_ctags_tagfile = '.tags'
let g:gutentags_file_list_command = {
      \ 'markers': {
      \ '.git': 'git ls-files',
      \ },
      \ }
let g:gutentags_generate_on_new = 1

Which theoretically should work fine since g:gutentags_file_list_command is set as far as I understood from the help.

This is my ~/.ctags file

-f .tags
-R
--fields=+l
--exclude=.svn
--exclude=.bundle
--exclude=.git
--exclude=node_modules
--exclude=log
--exclude=tmp
--exclude=*.pyc
--exclude=*.pyo
--exclude=*.min.*

--langdef=js
--langmap=js:.js
--langmap=js:+.jsx

--regex-js=/[ \t.]([A-Z][A-Z0-9._$]+)[ \t]*[=:][ \t]*([0-9"'\[\{]|null)/\1/n,constant/
--regex-js=/\.([A-Za-z0-9._$]+)[ \t]*=[ \t]*\{/\1/o,object/
--regex-js=/['"]*([A-Za-z0-9_$]+)['"]*[ \t]*:[ \t]*\{/\1/o,object/
--regex-js=/([A-Za-z0-9._$]+)\[["']([A-Za-z0-9_$]+)["']\][ \t]*=[ \t]*\{/\1\.\2/o,object/
--regex-js=/([A-Za-z0-9._$]+)[ \t]*=[ \t]*\(function\(\)/\1/c,class/
--regex-js=/['"]*([A-Za-z0-9_$]+)['"]*:[ \t]*\(function\(\)/\1/c,class/
--regex-js=/class[ \t]+([A-Za-z0-9._$]+)[ \t]*/\1/c,class/
--regex-js=/([A-Za-z$][A-Za-z0-9_$()]+)[ \t]*=[ \t]*[Rr]eact.createClass[ \t]*\(/\1/c,class/
--regex-js=/([A-Z][A-Za-z0-9_$]+)[ \t]*=[ \t]*[A-Za-z0-9_$]*[ \t]*[{(]/\1/c,class/
--regex-js=/([A-Z][A-Za-z0-9_$]+)[ \t]*:[ \t]*[A-Za-z0-9_$]*[ \t]*[{(]/\1/c,class/
--regex-js=/([A-Za-z$][A-Za-z0-9_$]+)[ \t]*=[ \t]*function[ \t]*\(/\1/f,function/
--regex-js=/(function)*[ \t]*([A-Za-z$_][A-Za-z0-9_$]+)[ \t]*\([^)]*\)[ \t]*\{/\2/f,function/
--regex-js=/['"]*([A-Za-z$][A-Za-z0-9_$]+)['"]*:[ \t]*function[ \t]*\(/\1/m,method/
--regex-js=/([A-Za-z0-9_$]+)\[["']([A-Za-z0-9_$]+)["']\][ \t]*=[ \t]*function[ \t]*\(/\2/m,method/

Right now if I open a generated minified file inside a folder that is ignored with git, ctags still runs. Also, my .tags file contains tags from these paths. Any idea how to make this work properly?

[ahmedelgabri]

With this setup, when gutentags generates tags, usually the ctags process eats my CPU. No clue why?

[ludovicchabant]

Hi!

I'm not sure if ctags eating all your CPU is normal or not -- you'd have to compare to running it yourself by hand, but I don't imagine it would be any different.

Your Gutentags config looks correct to me, so I'm not sure what's wrong... you could do the following:

• Check if Gutentags leaves a tags.files file behind, that would be the list of files it would pass to Ctags to process (you may want to comment out the trap line in the plat/unix/update_ctags.sh script, since it will delete that file on exit)
• Enable debug tracing in Gutentags (:call gutentags#toggletrace() or :let g:gutentags_trace = 1) and see, in Vim's messages and in the generated .log file (next to your tags file) what Gutentags is doing, and if it looks OK (for example, in the log file, you'll get the complete command line it uses to call Ctags)

[ahmedelgabri]

Ok so I have been monitoring this for more than a week now, I even tried a different ctags plugin & it's still the same issue.

ctags always is hitting 90+% CPU so it's not gutentags issue, the codebase I'm working on has around 10k files which shouldn't be a big deal, any clues, tips, etc... on how to find a solution for this? or if this is normal even?

[ludovicchabant]

On my machine ctags usually stays around 80% but that's for smaller codebases. I'd say it's more or less normal, but I'll let others post their experience just in case.

[YodaEmbedding]

Another possibility (using ripgrep) is:

let g:gutentags_file_list_command = 'rg --files'

[ludovicchabant]

Yeah, although I imagine that the cost of listing files with git ls-files vs. rg --files might be minimal compared to the cost of ctags scanning all those files?

But other than that, I don't really have any other pointers -- ctags will take a whole bunch of CPU on one core, but with multi-core machines it's not so noticeable. At work I also deal with a codebase with tens of thousands of files, but since I have a 16-core hyper-threaded machine, I don't pay attention :) (although it does take several minutes to re-scan). I'm not sure there's anything we can do, unless someone has other tricks?

[YodaEmbedding]

Could you do incremental updates? Check the datestamps of a folder and only refresh that folder's ctags if the folder has been modified since the last update. Then copy all the ctags for the various folders into one file. This should only be done for the immediate contents of the directory (subdirectories handle their own ctags).

This sort of functionality is probably best implemented upstream as a flag, e.g. ctags --incremental. My suggestion is more of a complicated bandaid.

EDIT: Or maybe just track individual files, generate a corresponding $PROJECT_ROOT/.ctags/path/to/file for them, and then concatenate everything together.

EDIT 2: I found some resources:

• Tim Pope's writeup
• MagicTags plugin

[ludovicchabant]

So it looks like MagicTags does the same thing Gutentags does already: when you save a file, it only re-indexes that one file, removes that file's entries from the tags file, and puts the new entries in. But that only helps when you save one file -- not when you need to reindex the codebase.

Generally speaking, I guess it would indeed make some sense to write a more sophisticated ctags executable that would maintain a database of the entire directory structure, so as to make proper incremental updates.... but of course, that's a whole new project of its own :)
(although if I had to work on a new ctags implementation, I would start with making it multi-threaded)

Making one tags file per file would probably break Vim, since codebases with 10k files would result in 10k tags files added to the tagfiles() list and I doubt that would scale well (although you never know I guess)... and even for folders, that can easily get in the thousands... but that's not even counting the problems of tracking files/directories coming and going, which is not trivial either to do efficiently... there are entire libs like inotify dedicated to this, actually.

In the meantime, you could give Universal Ctags to see if it's any better?

[YodaEmbedding]

I am on Universal Ctags. I haven't personally experienced these performance issues since I'm only working with a pretty small codebase at the moment.

[alphaCTzo7G]

This works pretty well... thanks..

[ludovicchabant]

Closing old issues.