How to get file path instead of virtual path in compilation_database.json?

[MartinSkold]

Hi!

I love this tool, but have not been able to use it in my companies monorepo due to the compilation_database.json which is generated contains mostly the symlinks.

Do you know any simple way to map the virtual path to the original file?

I'll give an example:

"file": "/usr/include/x86_64-linux-gnu/path/to/file.h",
    "arguments": [
...
    "-isystem",
          "bazel-out/linux_host_gcc-opt/bin/external/gtest-1.10.0/googlemock",
...

"file": "/usr/include/x86_64-linux-gnu/path/to/another_file.h",
    "arguments": [
...
    "-isystem",
          "-Ibazel-out/linux_host_gcc-opt/some/path/_virtual_includes/path",
...
    "-isystem",
          "-Ibazel-out/linux_host_gcc-opt/some/path/_virtual_includes/path/target_name_created_in_virtual_path_that_does_not_exist_in_repo",
...

I found some way to "fix" some of the paths by switching from strip_include_prefix = "include" to includes = ["include"], but it doesn't solve all the issues.

Have you run into the same issue before, or know how to extract the info to do the inverse mapping so I can replace all the virtual include paths?

Unfurtunately, sed replace is not enough

sed -i -e 's/bazel-out\/linux_host_gcc-opt\/bin\///g' compile_commands.json
sed -i -e 's/_virtual_includes\///g' compile_commands.json

[cpsauer]

Hey, Martin! Thanks for your kind words. Glad to hear it's addressing a need for you, too :)

Definitely something we've seen and heard about before. Had you already seen #102 for the strip_include_prefix -> includes stuff? If not, could I ask you to give it a read?

I think the easiest quick fix here is running a quick build for the target you care about (making sure the flags are the same). That'll create the _virtual_includes links for anything those files depend on. Would that work for your use case? (As in #102, strip_include_prefixes that can't be converted to includes don't generally have a find and replace equivalent, I don't think, sadly. Please do tell me if you have any ideas, though!)

Even if that's the best we can do, I want to make sure that we're at least automatically telling people that they should consider running a build in this case. Did the tool print out warnings about missing generated files on the first run (before cache)? If not, we should change that. And if so, we should generalize the warning to more clearly cover these.

Thanks,
Chris

P.S. Can I ask what kinds of projects--and company--you're using it for? I always love to hear and understand where it's needed.

[cpsauer]

^ #121 is the same underlying thing, also worth a quick skim.

As there, if you're feeling especially crafty, I'd love it if you'd be willing to work on some code as a PR, quickly skimming the output for missing _virtual_includes and then issuing a warning that the person should run a build or switch to includes. We'll probably want to cache the existence check--for speed.

[MartinSkold]

Thank you @cpsauer

I guess that I need to start with just fixing the includes in the whole repository, so we don't use strip_include_prefix = "include" as I mentioned in the initial post, but I think there might be more things.

I do get a error message for one file, but I think it is the first it finds.

>>> A source file you compile doesn't (yet) exist: external/path/to/file.cpp
    It's probably a generated file, and you haven't yet run a build to generate it.
    That's OK; your code doesn't even have to compile for this tool to work.
    If you can, though, you might want to run a build of your code.
        That way everything is generated, browsable and indexed for autocomplete.
    However, if you have *already* built your code, and generated the missing file...
        Please make sure you're supplying this tool with the same flags you use to build.
        You can either use a refresh_compile_commands rule or the special -- syntax. Please see the README.
        [Supplying flags normally won't work. That just causes this tool to be built with those flags.]
    Continuing gracefully...

Was that the message you ment?

---

I am trying to use it at Tobii, but I haven't really succeded since it requires that we update most of the build files. It is a bit cumbersome to change every target, since there are so many. The hard part is to get priority on it, so we will see how it goes. If I am ever able to fix these issues.

The goal that I am trying to achieve is to use it for static code analysis, but also clangd plugin in vscode. Since it is multiplatform and multiple targets, a compliation database is useful when exploring the code base. If you generate the compilation database and use clangd, you could click on a definition and you would be taken to the correct implementation used for your target/platform.

[cpsauer]

Hey, hey! Happened to see this before I went to bed.

How about trying a quick build then! As above (and in the warning message), I think that'll build the symlinks you're missing and resolve things.

[We should still add a separate warning, since you can trip the _virtual_includes case without tripping the generated-source warning.]

[cpsauer]

Cool! Tobii the eye tracking company, right? I was just playing a game that was integrated with you guys: Blades of the Shogun. And have used your hardware in Stanford Psych experiments before, I'm pretty sure.

Anything I should know about what analysis you're contemplating running? Totally get the clangd use case. (That's what I originally built this for, as you probably know.)

[MartinSkold]

That is fantastic to hear that you have tried Tobii's products :) I have not played that specific game. Did it add anything for you in the immersion? I am still relatively new here compared to many other people, but for gaming, have you tried PSVR2 yet? Horizon is quite fun!

---

For static code analysis, that is still under investigation. I found quite a few foss tools that I wanted to test that relied on having a compilation database. Ofc, I could try it out in a test repository, but it is always more interesting to see what you have in your own repository ;)

---

Sorry, I don't think I understood exactly what you ment with a quick build. Did you mean to just rebuild it without a bazel clean in between?

[cpsauer]

As in bazel build the targets you care about!

Should build those symlinks--and then clangd will pick them up on reload (usually takes a file edit). No cleans. That'll delete the symlinks :)

(Sorry for incomplet reply; need to crash and figured I should prioritize trying to help.)

[Attempt3035]

Hi all! I've been using this tool on some of my projects, however I've got some pretty deep multi-repo structures. @cpsauer You're right in that building the targets fixes the symlinks, but that really isn't a great solution for these projects, as we have stacks of dependencies, even two or more layers deep, and i've found I've gotta manually specify and build each target, not just the top layer, or I end up with a ton of _virtual_includes with symlinks that don't exist 😝 Absolute pain in the butt!

Of course, this is much more a Bazel problem than with this tool, but I'd really like to help work on a solution or workaround we could use! I'm temporarily using includes = [] instead of strip_include_prefix as it's the strip_include_prefix that ends up putting things in _virtual_includes folders. I'd really like to avoid doing that long term though, as that means headers end up polluting the global space I believe? I'd like to ensure I'm keeping every package as private and scoped as possible!

Have there been any further thoughts on this? So far, I've wondered about:

1. Can we trigger a specific build of each target starting from the bottom of the dependency tree? Would these then be used by other packages (so we aren't adding to build time) or does it end up getting built into different places? I don't really understand the crux of why Bazel has this behaviour!
2. Can we run a post process path substitution of sorts where we are substituting flags to _virtual_includes folders with the paths via external// because these are symlinked back to the source and are there before each dep is built. (This seems hard because I believe we loose info when it ends up in a _virtual_includes folder. We don't know what directories were trimmed out for it to get there. Maybe we are able to get this info elsewhere?)

I'm sure there's a lot I'm missing, like how this might work with generated files and where they get generated to when only the top level target is built, but I'd really like any input, pointers or where to even start? Is a solution feasable? Is there something I can work on? This would be super helpful to my projects, (essential tbh...) and I'm sure it could help others too!

[Attempt3035]

Secondly, and I know this is probably a much broader question / issue, but, compile commands seem to always contain references to headers in the bazel sandbox, which makes sense (as that's where it's building from), but it also means all the header links go to copied files, not originals. Is it within any sort of possibility to have these mapped back to originals? Is this level of information available out of Bazel or would this constitute an impossible sort of reverse engineering task?

Edit:
My case is special, I just realised. For most, the paths would only be weird when using virtual includes (like strip_include_prefix etc). But as I've set up multiple projects depending on eachother, they are included as a dep and then a git or local override. That's probably not the best way to do it for a multi-repo Bazel project but I actually haven't found any other viable alternative. Anyway, as they are brought in as external deps, that's why the paths are not to originals, understandable.

Edit 2:
I've moved to running a multirepo setup where each repo is symlinked into where it needs to be. Each has a BUILD.bazel and the whole thing is treated as a monorepo as far as bazel build setup. I've come to realise bazel isn't really optimised for multirepo and makes things more complicated than necessary if everything is treated as a dep. This tool works much better this way too, but I still have the same issues with virtual includes🤪

[cpsauer]

Hey, guys!

I put up a PR that should fix over at bazelbuild/bazel#20540. Any amount you'd like to chime in over there to encourage merging would be much appreciated :)

#140 and #147 and #133 are the same underlying issue, I think.

[MartinSkold]

Hey, guys!

I put up a PR that should fix over at bazelbuild/bazel#20540. Any amount you'd like to chime in over there to encourage merging would be much appreciated :)

#140 and #147 and #133 are the same underlying issue, I think.

Thanks for creating that PR on the bazel repository, it might solve the issues I currently have. I will see if I get time to build bazel locally with your patch.

[cpsauer]

The fix has finally made it into the bazel mainline!

Thanks all for helping track this down and get it in--should be released as part of Bazel 7.1 and the next rolling release!

[MartinSkold]

Thanks for all the engagement! I appreciate it!

[cpsauer]

Thanks for appreciating! Yay team :)