[cdac] cdac-build-tool #100650

lambdageek · 2024-04-04T18:47:24Z

Contributes to #99298

cDAC Build Tool

Summary

The purpose of cdac-build-tool is to generate a .c file that contains a JSON cDAC contract descriptor.

It works by processing one or more object files containing data descriptors and zero or more text
files that specify contracts.

Running

% cdac-build-tool compose [-v] -o contractdescriptor.c -c contracts.txt datadescriptor.o

.NET runtime build integration

cdac-build-tool is meant to run as a CMake custom command.
It consumes a target platform object file and emits a C source
file that contains a JSON contract descriptor. The C source
is the included in the normal build and link steps to create the runtime.

The contract descriptor source file depends on contract-aux-data.c which is a source file that contains
the definitions of the "indirect pointer data" that is referenced by the data descriptor. This is typically the addresses of important global variables in the runtime.
Constants and build flags are embedded directly in the JSON payload.

Multiple data descriptor source files may be specified (for example if they are produced by different components of the runtime, or by different source languages). The final JSON payload will be a composition of all the data descriptors.

Multiple contracts text files may be specified. This may be useful if some contracts are conditionally included (for example if they are platform-specific). The final JSON payload will be a composition of all the contracts files.

flowchart TB
  headers("runtime headers")
  data_header("datadescriptor.h")
  data_src("datadescriptor.c")
  compile_data["clang"]
  data_obj("datadescriptor.o")
  contracts("contracts.txt")
  globals("contractpointerdata.c")
  build[["cdac-build-tool"]]
  descriptor_src("contractdescriptor.c")
  vm("runtime sources")
  compile_runtime["clang"]
  runtime_lib(["libcoreclr.so"])

  headers -.-> data_src
  headers ~~~ data_header
  data_header -.-> data_src
  headers -.-> globals
  headers -.-> vm
  data_src --> compile_data --> data_obj --> build
  contracts ---> build
  build --> descriptor_src
  descriptor_src --> compile_runtime
  data_header -.-> globals ----> compile_runtime
  vm ----> compile_runtime --> runtime_lib

We don't parse them yet, however

Store the magic as a uint64_t so that it will follow the platform endianness. Store endmagic as bytes so that it directly follows the name pool - and fix the endmagic check not to look at the endianness

…tractDescriptor

Co-authored-by: Elinor Fung <elfung@microsoft.com>

src/coreclr/debug/runtimeinfo/datadescriptor.cpp

src/coreclr/tools/cdac-build-tool/DataDescriptorModel.cs

src/coreclr/tools/cdac-build-tool/JsonConverter/GlobalModelJsonConverter.cs

src/coreclr/tools/cdac-build-tool/cdac-build-tool.csproj

…nConverter.cs Co-authored-by: Elinor Fung <elfung@microsoft.com>

lambdageek · 2024-04-18T23:25:46Z

All the windows build lanes are failing with something like:

.dotnet\sdk\9.0.100-preview.3.24204.13\Microsoft.Common.CurrentVersion.targets(4806,5): error MSB3026: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not copy "D:\a\_work\1\s\artifacts\obj\ILLink.Tasks\Debug\net9.0\ILLink.Tasks.dll" to "D:\a\_work\1\s\artifacts\bin\ILLink.Tasks\Debug\net9.0\ILLink.Tasks.dll". Beginning retry 1 in 1000ms. The process cannot access the file 'D:\a\_work\1\s\artifacts\bin\ILLink.Tasks\Debug\net9.0\ILLink.Tasks.dll' because it is being used by another process. The file is locked by: ".NET Host (5476)"

which looks like we're running the in-tree artifacts/bin/.../ILLink.Tasks.dll at the same time as we're building it (a second time?) I guess the obvious suspect is the compilation of cdac-build-tool, although i'm unclear why that would use the ILLink tasks from the in-tree build

elinor-fung · 2024-04-19T05:03:17Z

I think _RequiresLiveILLink is ending up true becasue EnableSingleFileAnalyzer is set to true in

runtime/eng/Analyzers.targets

Lines 15 to 18 in 3cea1ee

    
               <EnableSingleFileAnalyzer Condition=" 
        
                 '$(EnableSingleFileAnalyzer)' == '' and 
        
                 '$(TargetFrameworkIdentifier)' == '.NETCoreApp' and 
        
                 '$(IsSourceProject)' == 'true'">true</EnableSingleFileAnalyzer>

Maybe we should set EnableSingleFileAnalyzer to false instead of the private-per-underscore-convention property - similar to what we do for illink tasks/analyzers themselves.

shushanhf · 2024-04-19T06:47:58Z

@lambdageek @elinor-fung

After this PR, there is an error during coreclr camke-config step by command

####linux-debian-AMD64 and loongarch64 are both failed !!!
./build-runtime.sh -debug -loongarch64 -nopgooptimize -skipmanaged

-- Looking for process_vm_readv
-- Looking for process_vm_readv - found
CMake Error at debug/runtimeinfo/CMakeLists.txt:57 (message):
  No cdac-build-tool set or does not exist


-- Configuring incomplete, errors occurred!
See also "/home/qiao/work_qiao/runtime/artifacts/obj/coreclr/linux.loongarch64.Debug/CMakeFiles/CMakeOutput.log".
See also "/home/qiao/work_qiao/runtime/artifacts/obj/coreclr/linux.loongarch64.Debug/CMakeFiles/CMakeError.log".

am11 · 2024-04-19T10:42:47Z

src/coreclr/debug/runtimeinfo/CMakeLists.txt

+
+set(GENERATED_CDAC_DESCRIPTOR_DIR "${CMAKE_CURRENT_BINARY_DIR}/cdac")
+set(CONTRACT_DESCRIPTOR_OUTPUT "${GENERATED_CDAC_DESCRIPTOR_DIR}/contract-descriptor.c")
+if("${CDAC_BUILD_TOOL_BINARY_PATH}" STREQUAL "" OR NOT EXISTS "${CDAC_BUILD_TOOL_BINARY_PATH}")


nit:

- "${CDAC_BUILD_TOOL_BINARY_PATH}" STREQUAL "" + NOT CDAC_BUILD_TOOL_BINARY_PATH

it checks both "not defined" and "evaluates to empty string".

am11 · 2024-04-19T11:03:40Z

src/coreclr/debug/runtimeinfo/CMakeLists.txt

+add_custom_command(
+    OUTPUT "${CONTRACT_DESCRIPTOR_OUTPUT}"
+    VERBATIM
+    COMMAND dotnet ${CDAC_BUILD_TOOL_BINARY_PATH} compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>


src/coreclr/build-runtime.sh direct invocation skips requiring dotnet/msbuild, which is necessary for new platform port work. This PR assumes dotnet/msbuild are always available. We should handle that case.

Secondly, can use published cdac-build-tool instead of assembly path so instead of requiring dotnet in PATH, it uses executable (AOT'd, singilefilehost, apphost -based doesn't matter)? If that's not possible (e.g. crossbuild case), then please pass DOTNET_HOST_PATH (set by SDK to the executing dotnet(1)) via cmakeargs as well.

Can we just make embedding the contract descriptor optional if someone is doing a build without msbuild?

If making it optional doesn't affect debugging, that's good. Second part is:

- <_CoreClrBuildArg Include="-cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=$(RuntimeBinDir)cdac-build-tool\cdac-build-tool.dll"" /> + <_CoreClrBuildArg Include="-cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH='$(RuntimeBinDir)cdac-build-tool/cdac-build-tool.dll'" -cmakeargs "-DCLR_DOTNET_HOST_PATH='$(DOTNET_HOST_PATH)'"" />

- COMMAND dotnet ${CDAC_BUILD_TOOL_BINARY_PATH} compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor> + COMMAND "${CLR_DOTNET_HOST_PATH}" "${CDAC_BUILD_TOOL_BINARY_PATH}" compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>

which will resolve to path/to/runtime/.dotnet/dotnet.

If making it optional doesn't affect debugging, that's good

in the future, it will make using something like dotnet-sos impossible

I don't understand what scenarios don't have a .NET sdk available. any new platform bringup will start with cross compiling from an existing platform with a .NET sdk and a cross compiler toolchain, right? that case should be handled by the cmake+msbuild infrastructure

I added #101297

- COMMAND dotnet ${CDAC_BUILD_TOOL_BINARY_PATH} compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor> + COMMAND "${CLR_DOTNET_HOST_PATH}" "${CDAC_BUILD_TOOL_BINARY_PATH}" compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>

I had something like this before and it broke source build because it was pointing to the repo root .dotnet/dotnet, not to artifacts/sb/.dotnet/dotnet. AFAICT it's fine to just call "dotnet" from the path. eng/commons/build.sh always adds the right SDK dir to the path - it's done by InitializeDotNetCli, and it is source-build aware

Update I'm educating myself over in dotnet/installer#19534 (comment)

I think previously I used something other than DOTNET_HOST_PATH

Updated #101297 to use DOTNET_HOST_PATH if it's set

# cDAC Build Tool ## Summary The purpose of `cdac-build-tool` is to generate a `.c` file that contains a JSON cDAC contract descriptor. It works by processing one or more object files containing data descriptors and zero or more text files that specify contracts. ## Running ```console % cdac-build-tool compose [-v] -o contractdescriptor.c -c contracts.txt datadescriptor.o ``` ## .NET runtime build integration `cdac-build-tool` is meant to run as a CMake custom command. It consumes a target platform object file and emits a C source file that contains a JSON contract descriptor. The C source is the included in the normal build and link steps to create the runtime. The contract descriptor source file depends on `contract-aux-data.c` which is a source file that contains the definitions of the "indirect pointer data" that is referenced by the data descriptor. This is typically the addresses of important global variables in the runtime. Constants and build flags are embedded directly in the JSON payload. Multiple data descriptor source files may be specified (for example if they are produced by different components of the runtime, or by different source languages). The final JSON payload will be a composition of all the data descriptors. Multiple contracts text files may be specified. This may be useful if some contracts are conditionally included (for example if they are platform-specific). The final JSON payload will be a composition of all the contracts files. ```mermaid flowchart TB headers("runtime headers") data_header("datadescriptor.h") data_src("datadescriptor.c") compile_data["clang"] data_obj("datadescriptor.o") contracts("contracts.txt") globals("contractpointerdata.c") build[["cdac-build-tool"]] descriptor_src("contractdescriptor.c") vm("runtime sources") compile_runtime["clang"] runtime_lib(["libcoreclr.so"]) headers -.-> data_src headers ~~~ data_header data_header -.-> data_src headers -.-> globals headers -.-> vm data_src --> compile_data --> data_obj --> build contracts ---> build build --> descriptor_src descriptor_src --> compile_runtime data_header -.-> globals ----> compile_runtime vm ----> compile_runtime --> runtime_lib ``` --- * add implementation note notes * add an emitter * read in the directory header * contract parsing * indirect pointer value support * move sample to tool dir * Take baselines from the docs/design/datacontracts/data dir We don't parse them yet, however * Add README * fix BE Store the magic as a uint64_t so that it will follow the platform endianness. Store endmagic as bytes so that it directly follows the name pool - and fix the endmagic check not to look at the endianness * hook up cdac-build-tool to the coreclr build; export DotNetRuntimeContractDescriptor * cleanup; add contracts.txt * add diagram to README * move implementation notes * better verbose output from ObjectFileScraper * turn off whole program optimizations for data-descriptor.obj On windows /GL creates object files that cdac-build-tool cannot read It's ok to do this because we don't ship data-descriptor.obj as part of the product - it's only used to generate the cDAC descriptor * C++-ify and add real Thread offsets * no C99 designated initializers in C++ until C++20 * build data descriptor after core runtime * fix gcc build * simplify ObjectFileScraper just read the whole file into memory * invoke 'dotnet cmake-build-tool.dll' instead of 'dotnet run --project' * clean up macro boilerplate * platform flags * turn off verbose output * can't use constexpr function in coreclr because debugreturn.h defines a `return` macro that expands to something that is not c++11 constexpr * Rename "aux data" to "pointer data" * rename "data-descriptor" to "datadescriptor" * simplify linking * cdac-build-tool don't build dotnet tool; turn on analyzers * rationalize naming; update docs; add some inline comments * renamce cdac.h to cdacoffsets.h * improve output: hex offsets; improved formatting * don't throw in ParseContracts; add line numbers to errors * change input format for contracts to jsonc * add custom JsonConverter instances for the compact json representation * simplify; bug fix - PointerDataCount include placeholder * one more set of feedback changes: simpler json converters * set _RequiresLiveILLink=false for cdac-build-tool.csproj fixes windows builds: error MSB3026: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not copy "D:\a\_work\1\s\artifacts\obj\ILLink.Tasks\Debug\net9.0\ILLink.Tasks.dll" to "D:\a\_work\1\s\artifacts\bin\ILLink.Tasks\Debug\net9.0\ILLink.Tasks.dll". Beginning retry 1 in 1000ms. The process cannot access the file 'D:\a\_work\1\s\artifacts\bin\ILLink.Tasks\Debug\net9.0\ILLink.Tasks.dll' because it is being used by another process. --------- Co-authored-by: Elinor Fung <elfung@microsoft.com> Co-authored-by: Aaron Robinson <arobins@microsoft.com>

lambdageek and others added 15 commits April 4, 2024 10:48

checkpoint

8518e9f

add old tooling notes

12aef6d

add an emitter

1eff4d0

checkpoint read in the directory header

22e217b

checkpoint plausible-looking offsets

dd9279d

checkpoint: field stuff is working

492af29

checkpoint json

459d124

contract parsing

28f8256

refactor

418739d

indirect pointer value support

1c1524d

move sample to tool dir

34e852e

Take baselines from the docs/design/datacontracts/data dir

e48e709

We don't parse them yet, however

Add README

0e8c60e

fix BE

cd4f0f0

Store the magic as a uint64_t so that it will follow the platform endianness. Store endmagic as bytes so that it directly follows the name pool - and fix the endmagic check not to look at the endianness

hook up cdac-build-tool to the coreclr build; export DotNetRuntimeCon…

b6b079e

…tractDescriptor

dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Apr 4, 2024

dotnet-policy-service bot assigned lambdageek Apr 4, 2024

lambdageek added area-Diagnostics-coreclr NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Apr 4, 2024

lambdageek and others added 2 commits April 4, 2024 15:14

try fix win-x86 build warning

a9b0d7f

paths

dbf0557

lambdageek force-pushed the cdac-contract-tool branch from 1d443ba to dbf0557 Compare April 4, 2024 22:45

lambdageek and others added 3 commits April 4, 2024 19:23

vebose

74433f1

fix typo; fixes win32

ea441b6

cleanup; add contracts.txt

787b836

lambdageek and others added 3 commits April 17, 2024 10:10

Apply suggestions from code review

e143998

Co-authored-by: Elinor Fung <elfung@microsoft.com>

simplify; bug fix - PointerDataCount include placeholder

08cf75d

fixup docs

0d4d40d

elinor-fung mentioned this pull request Apr 17, 2024

[cdac] Read contract descriptor from target #101208

Merged

Merge remote-tracking branch 'origin/main' into cdac-contract-tool

94c56ea

elinor-fung approved these changes Apr 18, 2024

View reviewed changes

src/coreclr/debug/runtimeinfo/datadescriptor.cpp Outdated Show resolved Hide resolved

src/coreclr/tools/cdac-build-tool/DataDescriptorModel.cs Outdated Show resolved Hide resolved

src/coreclr/tools/cdac-build-tool/JsonConverter/GlobalModelJsonConverter.cs Outdated Show resolved Hide resolved

elinor-fung reviewed Apr 18, 2024

View reviewed changes

src/coreclr/tools/cdac-build-tool/cdac-build-tool.csproj Outdated Show resolved Hide resolved

Update src/coreclr/tools/cdac-build-tool/JsonConverter/GlobalModelJso…

3d49b70

…nConverter.cs Co-authored-by: Elinor Fung <elfung@microsoft.com>

lambdageek mentioned this pull request Apr 18, 2024

[cdac-build-tool] Take baseline as a command line argument #101257

Open

This was referenced Apr 18, 2024

[wasm] System.Text.Json.Tests running out of memory #98578

Closed

WASM test failing with OOM #100111

Closed

Debug assertion failure in System.Net.Http.HttpConnection.SendAsync #100616

Closed

one more set of feedback changes

f845c41

maybe no live ILLink ?

64430cf

build-analysis bot mentioned this pull request Apr 19, 2024

[8.0] HttpClient test failure - attempting to access disposed object 'ConnectedStreams' #96797

Closed

lambdageek merged commit 4abe399 into dotnet:main Apr 19, 2024
149 of 155 checks passed

shushanhf mentioned this pull request Apr 19, 2024

Workaround for build failure in cdacreader when EmitCompilerGeneratedFiles=true #101279

Merged

am11 mentioned this pull request Apr 19, 2024

Adjust the calleeSavedRegs on top frame for LoongArch64/RISCV64 #100962

Merged

am11 reviewed Apr 19, 2024

View reviewed changes

akoeplinger mentioned this pull request Apr 19, 2024

[main] Update dependencies from dotnet/sdk dotnet/installer#19534

Merged

lambdageek mentioned this pull request Apr 19, 2024

[cdac] link a stub contract descriptor if cdac-build-tool is not available #101297

Merged

github-actions bot locked and limited conversation to collaborators May 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[cdac] cdac-build-tool #100650

[cdac] cdac-build-tool #100650

lambdageek commented Apr 4, 2024 •

edited

Loading

lambdageek commented Apr 18, 2024

elinor-fung commented Apr 19, 2024

shushanhf commented Apr 19, 2024 •

edited

Loading

am11 Apr 19, 2024

am11 Apr 19, 2024 •

edited

Loading

lambdageek Apr 19, 2024

am11 Apr 19, 2024 •

edited

Loading

lambdageek Apr 19, 2024 •

edited

Loading

lambdageek Apr 19, 2024

lambdageek Apr 19, 2024

[cdac] cdac-build-tool #100650

[cdac] cdac-build-tool #100650

Conversation

lambdageek commented Apr 4, 2024 • edited Loading

cDAC Build Tool

Summary

Running

.NET runtime build integration

lambdageek commented Apr 18, 2024

elinor-fung commented Apr 19, 2024

shushanhf commented Apr 19, 2024 • edited Loading

am11 Apr 19, 2024

Choose a reason for hiding this comment

am11 Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

lambdageek Apr 19, 2024

Choose a reason for hiding this comment

am11 Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

lambdageek Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

lambdageek Apr 19, 2024

Choose a reason for hiding this comment

lambdageek Apr 19, 2024

Choose a reason for hiding this comment

lambdageek commented Apr 4, 2024 •

edited

Loading

shushanhf commented Apr 19, 2024 •

edited

Loading

am11 Apr 19, 2024 •

edited

Loading

am11 Apr 19, 2024 •

edited

Loading

lambdageek Apr 19, 2024 •

edited

Loading