RFE: Improve rootDirectory detection

[har7an]

Hello,

First: Thanks for this amazing project!
Second: I'm using texlab with nvim-lspconfig to build a multi-file project with tectonic. My problem is that I do not want to configure the texlab rootDirectory setting globally in nvim, because I'm working on multiple documents. However, the root directory detection doesn't work correctly otherwise.

I've looked at the code and it seems that when rootDirectory isn't defined (None), it uses a custom fallback in all the necessary locations, which mostly defaults to using the files parent directory. While this allows me to build all files and get diagnostics, it doesn't pick up my custom latexindent config. Would you be interested in an "improved" rootDirectory detection, or is this better left to the editors using LSP?

[pfoerster]

@har7an
Thanks for the kind words.

Have you tried out #823? It changes a lot of the internals in the project detection method. In particular, rootDirectory is now relative to the main directory instead (e. g. if you have a src folder, it needs to be .. instead). This way, texlab does not rely on the working directory of the editor anymore, which is very prone to errors.

Would you be interested in an "improved" rootDirectory detection

Definitely, which ideas do you have in mind?

In the case of tectonic, we could also parse the Tectonic.toml manifest (there is even a crate for that) and use that to discover the project files.

[har7an]

Have you tried out #823?

No I haven't, but I can do that of course! Impressive changeset.

Definitely, which ideas do you have in mind?

One could do something similar to nvim: Walk up the file tree and check, in a pre-defined order, for the existence of special files (i.e. .git, Tectonic.toml, ...). Although I guess this brings its own heap of problems in special cases where people e.g. have multiple Tectonic.toml in one git project.

For the time being, I have found this neat fix for nvim users:

require("lspconfig").texlab.setup {
    -- ...
    on_new_config = function(new_config, new_root_dir)
        new_config.settings.texlab.rootDirectory = new_root_dir
    end,
}

[pfoerster]

One could do something similar to nvim: Walk up the file tree and check, in a pre-defined order, for the existence of special files (i.e. .git, Tectonic.toml, ...)

Yeah, I have thought about this one too. We could extend this list to

• Tectonic.toml
• .latexmkrc
• .chktexrc
• latexindent config files
• .git
• (and maybe even src/source)

For .latexmkrc and .chktexrc we would need a check though to not consider the global config inside the home directory but Tectonic.toml should be pretty safe.

Although I guess this brings its own heap of problems in special cases where people e.g. have multiple Tectonic.toml in one git project

This should be no problem anymore with #823. Root folder detection is now per document instead of globally.

No I haven't, but I can do that of course!

Sure, that would be awesome. I am particulary interested if there are any regressions that the tests do not catch. The performance should be mostly equal but completion should be much snappier.

[har7an]

#823 definitely improves performance! Unfortunately, it doesn't resolve the rootDirectory issue on its own. It still requires the nvim config I posted above.

Yeah, I have thought about this one too. We could extend this list to

Sounds reasonable, one of those should surely work.

[pfoerster]

Unfortunately, it doesn't resolve the rootDirectory issue on its own. It still requires the nvim config I posted above.

@har7an Can you give #823 a try again, please? It does now implement the strategy mentioned above.

[xlucn]

I am also having issues with the new version 5.0.0.

It seams that .git directory detection has too high priority. For example, I have this git repo structure:

repo/
- .git/
- dir1/
  - main.tex
- dir2/
  - main.tex

Currently, if I edit dir1/main.tex or dir2/main.tex, texlab seems to set root directory to repo/. But I want the root directory to be either dir1 or dir2, depending on which file I am editing. (Update: set rootDirectory = '.' explicitly does not seem to change anything)

So, does the current behavior need improvements? Or, what workaround can I use to achieve what I want? Thanks in advance.

[leon-richardt]

Transferring a comment from #838 (comment) by @xlucn:

---

[...]

What I had in mind is more automatic solution. For example, if current file has \begin{document} and \end{document}, then use the folder of current file as root directory. In other words, I hope texlab can find the main document and use its parent folder as root directory. This behavior is intuitive IMHO. (Maybe I should continue the discussion in #826 or in a new issue?)

Here is the reason of the proposal. Texlab is, I guess, executing latexmk from the root directory. So, if the root directory is different from the folder where the main document is, not only the output files are in the wrong place, but texlab also cannot find some included files such as included graphics or tex files. Therefore, I think it's better to determine root directory by the main document, not .git folder or some configuration files.

---

I think your proposal could work as a "more reasonable" default but I'm not sure it will catch all use cases either. It is possible to have multiple begin{document}/\end{document} pairs in a project, e.g., as in the subfiles example on Overleaf. This is the only example I can think of right now but there might be more.

Still, I think your approach would work very well as a default.¹ Maybe it can be combined with #838 for those rare cases where it would fail. On a related note, I'm not sure if implementing a general, always-correct project detection is even possible. TeX is so flexible in the project structures it allows that a general detection must be very hard at least, if not impossible.

Footnotes

1. Although I don't know how feasible opening and parsing many files would be; this is something the maintainers might be able to shed light on. ↩

[xlucn]

@leon-richardt thanks for the reply (in both threads) and transferring the comment.

I looked at vimtex's strategy of dealing with multi file project. Maybe that's helpful for texlab devs, especially the "Directory scan" section. They seems also to cover the subfiles case.

[pfoerster]

In other words, I hope texlab can find the main document and use its parent folder as root directory.

This was the usual behavior for version 4.x if you set the rootDirectory to null. texlab iteratively checks the parent directory until it finds a suitable parent document. This works relatively fine if you always compile from the same directory as the main file.

However, it does not work if you compile from another directory. Some people like to use a src directory so we needed a way to tell texlab from which directory the document is compiled from. To handle this case, we introduced the rootDirectory setting. In older versions it was based on the current working dir (which is a bad idea since it highly depends on the editor), 5.0.0 makes it relative to the parent document.

Although I don't know how feasible opening and parsing many files would be; this is something the maintainers might be able to shed light on

In my experience, this approach has worked fine in the past (and it still does; salsa might also help us discarding unused parse trees in this case).

Maybe it can be combined with #838 for those rare cases where it would fail. On a related note, I'm not sure if implementing a general, always-correct project detection is even possible.

A simple (and predictable) solution would be the following approach:

For each new/changed document, perform the following steps:

1. Recursively walk up the directory tree and open the TeX documents on the way up. Stop traversal if we see a texlabroot file or we reach the file system root
2. For each opened TeX document, we check if it is a parent of the current document. Use the texlabroot directory if available; otherwise use the document directory. If we found a document, then we can stop looking for a parent and resolve the includes of the project.
3. Load the .aux files (if any) using the texlab.auxDirectory setting by treating the path relative to the root directory. If not set, use the root directory instead.

Further, we would remove special files like .latexmkrc, .chktex or .git directories from the project detection algorithm (maybe with the exception of Tectonic.toml since it splits up document environment over multiple files; we can use the tectonic_docmodel crate for this purpose).

@xlucn @leon-richardt What do you think?

[leon-richardt]

@xlucn @leon-richardt What do you think?

I don't know how valuable my opinion is as I'm sure you have put much more thought into this than I have. But your suggestion sounds good to me; it would certainly solve my scenario!

Further, we would remove special files like .latexmkrc, .chktex or .git directories from the project detection algorithm (maybe with the exception of Tectonic.toml since it splits up document environment over multiple files; we can use the tectonic_docmodel crate for this purpose).

I believe this would be a wise decision as well so as not to rely on assumptions about a user's project structure 👍

[xlucn]

Seems to cover my use case, thanks!

Just one thing, if a non-global .latexmkrc/latexmkrc file exists, doesn't it mean the user wants to execute latexmk in the same directory? That seems to indicate the root directory the user desires (If root directory IS the directory to execute the build command like latexmk, as I understand).

[pfoerster]

Just one thing, if a non-global .latexmkrc/latexmkrc file exists, doesn't it mean the user wants to execute latexmk in the same directory?

Generally yes, but not always. Consider the .latexmkrc file in the home directory, which can be used to change the global latexmk options. In this case, the file does not necessarily indicate a project.

@leon-richardt @xlucn I have a created a new PR (#839), which implements the algorithm in #826 (comment).

[xlucn]

Consider the .latexmkrc file in the home directory

Maybe the "non-global" word is not accurate enough. By that I refer to (using texts in latexmk manpage") "the RC file in the current working directory" except "the system RC file" or "the user's RC file". $HOME/.latexmkrc is "the user's RC file", thus don't use it as project root.

Also, it's similar to (but not entirely) common alg to determine project root based on git repository. The user's home is normally excluded, even if there is a $HOME/.git/ folder.

[har7an]

Sorry for not responding for so long.

I tested #839 and the project root detection works like a charm now for the project I initially discovered this on. :)

Good job!