OpenTelemetry C++ SDK build with vcpkg / OTLP-gRPC: 260 symbols MAX_PATH limitation

[maxgolov]

This is NOT OpenTelemetry C++ SDK issue. But rather a build issue with ninja, described here:

ninja-build/ninja#1900

Problem Description

Imagine a project that builds OpenTelemetry C++ SDK as a dependent git submodule in its own build tree. OpenTelemetry C++ SDK on Windows is easy to build using vcpkg. However, there is a file path length limitation of 260 symbols that is coming into play here if you are building with OTLP+gRPC enabled.

The following layout:
C:\agent\worker\MyInstrumentationFrameworkProject\third_party\opentelemetry-cpp\tools\vcpkg\buildtrees\grpc

fails to build miserably while trying to install OTLP-gRPC on Windows with the following error:

ninja: error: Stat(C:/agent/worker/MyInstrumentationFrameworkProject/third_party/opentelemetry-cpp/tools/vcpkg/buildtrees/grpc/src/2948941d75-78cfc6f1b1.clean/src/core/ext/upbdefs-generated/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.upbdefs.c): Filename longer than 260 characters

Since it is rather hard to predict where the source code is going to be located, especially if it gets pulled into another project built somewhere on a build pipeline as a submodule... we do not have much options. We may try to assume that a custom %TEMP% variable is set to short path. And we host vcpkg build tree inside %TEMP% rather than hosting it inside the tools/vcpkg directory? That may work. However, this approach has some security implications. Ideally the full build has to be isolated in a single work tree, without needing any external directories, especially external temporary directories which may be harder to isolate.

Immediate recommendation we can give that the origin name of opentelemetry working directory MUST BE shorter than 260 - 5 - strlen("/tools/vcpkg/buildtrees/grpc/src/2948941d75-78cfc6f1b1.clean/src/core/ext/upbdefs-generated/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.upbdefs.c"), that is no longer than 71 symbols.

In the above example - the path C:/agent/worker/MyInstrumentationFrameworkProject/third_party/opentelemetry-cpp/ is already too long. As it will fail vcpkg install gRPC:x64-windows installation steps within the same build tree, because it is exceeding the allowed base path length limit. There could be other issues coming into play later on: as new extensions could be added to gRPC source, there is no such limitation on Unix OS, and filenames could be even longer than the example above.

Quick recap - TL;DR

Don't use names that are "too long" as base path for building OpenTelemetry C++ SDK on Windows.

Use something short, like C:\otel\ - is a good name. And then the vcpkg gRPC build required for OTLP - passes okay. 😁

[maxgolov]

At this time I think we should capture this detail somewhere in Building OpenTelemetry for Windows document.. OR MAYBE an option that allows to host vcpkg outside of build tree, using shorter path.

[meastp]

@maxgolov how about using vcpkg's manifests feature (a vcpkg.json file describing dependencies)? And optionally a custom registry if there are packages that aren't in the official vcpkg registry (a custom git repo with custom ports)?

When using manifests, the packages are built in the vcpkg directory (used via vcpkg.exe integrate install) which could be in e.g. C:\vcpkg to have space for long file paths. (This for example is the path where Azure Hosted Windows Agents have pre-installed vcpkg)

[maxgolov]

@meastp - it should indeed work fine for the usual build pipelines, like Azure Hosted agents or GitHub Action runners. Maybe that is the best option. I think what we do with tools/vcpkg was not be the best option after all.. OR at least we should add some build parameter - environment variable that would prefer vcpkg from short path, like C:\vcpkg that you mentioned, instead of using vcpkg from submodule. Maybe we can simply set VCPKG_ROOT=C:\vcpkg as build environment variable. I need to try it out.

[meastp]

@maxgolov If you use vcpkg.exe integrate install then you don't need to communicate where your vcpkg installation is.

I think it would be a better option to either use dependencies as submodules etc. OR use vcpkg with manifests (and custom registries if needed) - not a combination. This is more in line with the documentation of vcpkg, and I think the experience would be smoother also :)

[maxgolov]

I think we can find creative ways of handling it without vcpkg instegrate install. My understanding we can still possibly pull the tools/vcpkg as submodule. But specify some shorter alternate path to build trees. That'd solve it, because the issue is not per se where vcpkg.exe is located, but how long is the path of individual file in a build tree. That same issue would even happen if we pull the entire gRPC in our own tree as submodule, in case if our own directory length is too long.

Presently the issue is only happening with gRPC, and gRPC is the main component that pulls so many other deps, that it's easier to handle the build and installation of it on Windows with vcpkg rather than pulling all these deps as submodules.

[meastp]

@maxgolov I'm not just talking about how to solve this specific issue, but using vcpkg more in line with how the documentation suggests it be used :) If you want to explore this more, I'm happy to help :)

Keep in mind that vcpkg.exe integrate install is done independently of building opentelemetry - it is a user-wide one-time operation.

However, I think that you want to support multiple different build systems, and this might be a least common denominator type of solution. Is this correct?

[maxgolov]

vcpkg.exe integrate install

This was, unfortunately, breaking some of my other projects. I personally do not do this on my dev box, i.e. prefer having separate build trees / separate snapshots of vcpkg per project. I have some thoughts about an easy fix for this exact issue, along the lines of using VCPKG_ROOT. It's a standard variable and we can start respecting it in the setup-buildtools.cmd script.