Print linking instructions for NativeLib=static #103960
Conversation
dotnet-policy-service
bot
added
the
community-contribution
These paths are constantly changing. It is ok as an example, but it is not something people can copy&paste into a robust build system. Would it be possible to send the linker command line into a file or something, and make the recipe that people copy&paste into their native build read it from that file? |
|
Yes, e.g. if user updates dotnet, they will get a newer ILCompiler package. Also user can clear the nuget cache. I think the copy-paste would still be needed since some of the non-path linker arguments are position-dependent and syntax vary by build system (cmake, meson. cargo). We could separate path-like arguments so e.g. they can copy them to a folder. Would that be better? |
|
Windows: Native AOT static lib linking instructionsFollow these steps to link your code with the statically compiled AOT binaries: :: Set some ILCompiler paths for convenience
set IlcSdkPath=C:\Users\adeel\.nuget\packages\runtime.win-arm64.microsoft.dotnet.ilcompiler\9.0.0-preview.6.24319.11\sdk\
set IlcFrameworkPath=C:\Users\adeel\.nuget\packages\runtime.win-arm64.microsoft.dotnet.ilcompiler\9.0.0-preview.6.24319.11\framework\
:: Compile your native code, e.g.
cl.exe -c glue.c /Foglue.o
:: Link your glue with AOT static libs
cl.exe glue.o /Fe:"myexe.exe" ^
"C:\Users\adeel\source\naotstatic\dist\\naotstatic.lib" ^
/DEF:"obj\Release\net9.0\win-arm64\native\naotstatic.def" ^
/LIBPATH:"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\lib\ARM64" ^
/LIBPATH:"C:\Program Files (x86)\Windows Kits\10\lib\10.0.22621.0\ucrt\arm64" ^
/LIBPATH:"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\arm64" ^
"%IlcSdkPath%bootstrapperdll.obj" ^
"%IlcSdkPath%Runtime.WorkstationGC.lib" ^
"%IlcSdkPath%eventpipe-disabled.lib" ^
"%IlcSdkPath%standalonegc-disabled.lib" ^
"%IlcSdkPath%System.Globalization.Native.Aot.lib" ^
"%IlcSdkPath%System.IO.Compression.Native.Aot.lib" ^
"advapi32.lib" ^
"bcrypt.lib" ^
"crypt32.lib" ^
"iphlpapi.lib" ^
"kernel32.lib" ^
"mswsock.lib" ^
"ncrypt.lib" ^
"normaliz.lib" ^
"ntdll.lib" ^
"ole32.lib" ^
"oleaut32.lib" ^
"secur32.lib" ^
"user32.lib" ^
"version.lib" ^
"ws2_32.lib" ^
/NOLOGO /MANIFEST:NO ^
/DEBUG ^
/INCREMENTAL:NO ^
/NATVIS:"C:\Users\adeel\.nuget\packages\microsoft.dotnet.ilcompiler\9.0.0-preview.6.24319.11\build\NativeAOT.natvis" ^
/IGNORE:4104 ^
/NODEFAULTLIB:libucrt.lib ^
/DEFAULTLIB:ucrt.lib ^
/OPT:REF ^
/OPT:ICF |
|
Regular copy&paste from .md file is only a small improvement over regular copy&paste from a log file. It makes a quick prototyping easier, but it is not something to use for real. I think we want a recipe that does not require regular copy&paste. It can look like what Xamarin iOS does. It has a custom target that saves the linker options in a form that the rest of the native build system expects. We can have a sample of such custom target e.g. for vanilla make. |
|
Tying this in with AaronRobinsonMSFT/DNNE#185, maybe we could consider generating a CMake module with an imported target that puts the pieces together so a simple |
It is a bit more than that, we have to add/remove a few things from the log capture to make it work. This makes the basic sample self-documenting.
Generally I like the idea. I think we can also write the paths and linker args as flat files (like cmake does in its cache directories under artifacts/obj) and that could be made importable in shell, batch, make and cmake etc. I think that will cover @jkotas' concern of making it reusable. Unfortunately, the floating nature package versions is something we can't do much about and I don't have better suggestion than "copy the required files to a safe location and link from there" (so if someone or some tool clears the nuget cache, their built can still be repeatable with the exact versions of binaries). |
|
For instance: Generated Similarly generated then the generated NativeAotLinkage.cmake cmake_minimum_required(VERSION 3.10)
set(NATIVEAOT_LINK_OPTIONS_FILE "${CMAKE_SOURCE_DIR}/linker_options.txt")
set(NATIVEAOT_OBJECTS_FILE "${CMAKE_SOURCE_DIR}/sources.txt")
# Read linker options from file
file(READ ${NATIVEAOT_LINK_OPTIONS_FILE} LINKER_OPTIONS_CONTENT)
# Split content into lines and process as needed
string(REGEX REPLACE "\n$" "" LINKER_OPTIONS_CONTENT "${LINKER_OPTIONS_CONTENT}")
string(REPLACE "\n" ";" NATIVEAOT_LINK_OPTIONS "${LINKER_OPTIONS_CONTENT}")
# Read sources from file
file(READ ${NATIVEAOT_OBJECTS_FILE} NATIVEAOT_OBJECTS_CONTENT)
# Split content into lines and process as needed
string(REGEX REPLACE "\n$" "" NATIVEAOT_OBJECTS_CONTENT "${NATIVEAOT_OBJECTS_CONTENT}")
string(REPLACE "\n" ";" NATIVEAOT_OBJECTS "${NATIVEAOT_OBJECTS_CONTENT}")
# Provide the properties to the parent CMakeLists.txt
set(NATIVEAOT_LINK_OPTIONS ${NATIVEAOT_LINK_OPTIONS} PARENT_SCOPE)
set(NATIVEAOT_OBJECTS ${NATIVEAOT_OBJECTS} PARENT_SCOPE)Sample usage in user code (which we can document in this md): cmake_minimum_required(VERSION 3.10)
project(MyAOTProject)
# Include the NativeAotLinkage.cmake script to set up properties
include(cmake/NativeAotLinkage.cmake)
# Create your target using the properties
add_executable(${PROJECT_NAME} ${NATIVEAOT_OBJECTS})
# Construct the link command with NATIVEAOT_LINK_OPTIONS
set(LINK_COMMAND "cc ${CMAKE_BINARY_DIR}/${PROJECT_NAME}.o -o ${PROJECT_NAME}")
foreach(opt IN LISTS NATIVEAOT_LINK_OPTIONS)
set(LINK_COMMAND "${LINK_COMMAND} ${opt}")
endforeach()
# Add a custom command to execute the link command
add_custom_command(
TARGET ${PROJECT_NAME} POST_BUILD
COMMAND ${LINK_COMMAND}
COMMENT "Linking AOT static libs"
)Similarly, for regular Unix make, # Path to linker options file (adjust path as needed)
LINKER_OPTIONS_FILE := linker_options.txt
# Path to sources file (adjust path as needed)
SOURCES_FILE := sources.txt
# Read linker options from file
LINKER_OPTIONS := $(shell cat $(LINKER_OPTIONS_FILE))
# Read sources from file
SOURCES := $(shell cat $(SOURCES_FILE))
# Function to compile each source file
define compile_source
$(1).o: $(1).c
cc -c $$< -o $$@
endef
# Create compile rules for each source file
$(foreach src,$(basename $(SOURCES)),$(eval $(call compile_source,$(src))))
# Define the main target (adjust as needed)
TARGET := myapp
$(TARGET): $(addsuffix .o,$(basename $(SOURCES)))
cc $$^ -o $$@ $(LINKER_OPTIONS)
# Phony target to clean up
.PHONY: clean
clean:
rm -f $(TARGET) $(addsuffix .o,$(basename $(SOURCES)))
# Export variables to be used in including Makefile
export LINKER_OPTIONS
export SOURCES
export TARGETwhich the user can include in their own setup: # Include NativeAotLinkage.mk
include path/to/NativeAotLinkage.mk
# Optionally define additional rules or variables
# ...
# Example usage of exported variables
$(TARGET): $(addsuffix .o,$(basename $(SOURCES)))
@echo "Building $(TARGET) with sources: $(SOURCES) and linker options: $(LINKER_OPTIONS)"
cc $^ -o $@ $(LINKER_OPTIONS)
# Define other rules as needed
# ...
# Phony target to clean up
.PHONY: clean
clean:
rm -f $(TARGET) $(addsuffix .o,$(basename $(SOURCES)))We can generate the files and put samples for shell, batch, makefile and cmake (and maybe vcxproj?) in markdown as needed. |
|
For CMake, I'd rather produce an imported object library that has the various options set, but something of that model would work as well. |
Generating |
|
@jkotas updated $ dotnet9 publish -p:PublishAot=true -p:NativeLib=Static -o dist --ucr *.csproj | tee
...
Linking helpers 'sources.txt' and 'linkoptions.txt' written to /Users/adeel/projects/naotstatic/dist/.
$ cat dist/sources.txt
/Users/adeel/projects/naotstatic/dist//naotstatic.a
/Users/adeel/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.6.24319.11/sdk/libbootstrapperdll.o
/Users/adeel/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.6.24319.11/sdk/libRuntime.WorkstationGC.a
/Users/adeel/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.6.24319.11/sdk/libeventpipe-disabled.a
/Users/adeel/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.6.24319.11/sdk/libstandalonegc-disabled.a
/Users/adeel/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.6.24319.11/sdk/libstdc++compat.a
/Users/adeel/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.6.24319.11/framework/libSystem.Native.a
/Users/adeel/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.6.24319.11/framework/libSystem.Globalization.Native.a
/Users/adeel/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.6.24319.11/framework/libSystem.IO.Compression.Native.a
/Users/adeel/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.6.24319.11/framework/libSystem.Net.Security.Native.a
/Users/adeel/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.6.24319.11/framework/libSystem.Security.Cryptography.Native.Apple.a
/Users/adeel/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.6.24319.11/framework/libSystem.Security.Cryptography.Native.OpenSsl.a
$ cat dist/linkoptions.txt
-Wl,-ld_classic
-Wl,-exported_symbols_list,"obj/Release/net9.0/osx-arm64/native/naotstatic.exports"
-Wl,-dead_strip
-gz=zlib
--target=arm64-apple-macos12.0
-g
-Wl,-rpath,'@executable_path'
-ldl
-lobjc
-lswiftCore
-lswiftFoundation
-lz
-licucore
-lm
-L/usr/lib/swift
-framework CoreFoundation
-framework CryptoKit
-framework Foundation
-framework Security
-framework GSSExample usage from command line: I'll add make/cmake sample once this is reviewed. |
| <FileAsset Include="$(PublishDir)\$(TargetName)$(NativeBinaryExt)" /> | ||
| <FileAsset Include="@(CustomLinkerArg)" Condition="Exists('%(CustomLinkerArg.Identity)') and '%(CustomLinkerArg.ExcludeFromStaticInstructions)' != 'true'" /> | ||
| <LinkOption Include="@(CustomLinkerArg)" Condition="!Exists('%(CustomLinkerArg.Identity)') and '%(CustomLinkerArg.ExcludeFromStaticInstructions)' != 'true'" /> |
There was a problem hiding this comment.
I wonder if it would make sense to lean into the linker argument separation we started for Xamarin/MAUI in #88294 (comment) and #89916. The separation should already be there on mac and we might need to put a couple things into different ItemGroup on Linux.
Basically instead of doing a heuristic filtering on CustomLinkerArg, we'd compose these lists from the ~5 properties and ItemGroups.
This heuristic would also work, but we'll probably end up with many ExcludeFromStaticInstructions exclusions (Linux has more switches that are in the unncessary/potentially problematic bucket - and even on mac, we'd probably want to exclude rpath, for example).
There was a problem hiding this comment.
I have tested it on both mac and linux and tried the resultant lists usage with eval cc myglue.o -o myexe $(cat dist/sources.txt) $(cat dist/linkoptions.txt) using this program. It links with clang on mac and linux and gcc on linux drivers and bfd linker on linux (lld runs into #70277 (comment), I'll investigate it separately). Both on linux and mac I'm getting:
Hello from C# Add!
Hello from C callback!
sum is: 3
These options are ok to make linked app just work. If they want to customize it further, like generating a static binary out of the NativeLib=Static, they will need to set StaticICULinking, StaticOpenSslLinking etc. and pass -static to link options. While we can't predict and automate for all kinds of use-cases (this is just a head-start for user), we can definitely minimize this set to bare minimum and filter out the rest to let user make their own choices.
The problem with separate lists is that LinkerArg is something used in the wild. Users have their own options set. That's why I thought of using this inference. If we are going to split them and document the "sources" vs. "options" msbuild items, that would be great, but as it stands, I couldn't think of better way.
There was a problem hiding this comment.
lld runs into #70277 (comment), I'll investigate it separately
I take it back. To use lld, we actually need to publish with -p:LinkerFlavor=lld (so we get -Wl,-T,"obj/Release/net9.0/linux-musl-arm64/native/sections.ld" in linkoptions.txt).
Note that if user published with LinkerFlavor=lld and then decided to use bfd to link with their own code, it gives syntax error (all linkers have seem to invented their own layout script format and there is no consensus). So they will need to just drop the -Wl,-T line.
There was a problem hiding this comment.
I wonder if it would make sense to lean into the linker argument separation we started for Xamarin/MAUI
I think this is a good point. Do we believe that Xamarin/MAUI can switch over to this scheme? They are the real-world use for static linking. If they are not, it suggests that this is not the best design. It would be best to avoid creating and maintaining multiple recipes for static linking.
There was a problem hiding this comment.
@jkotas do we have a publicaly visible link where they are using NativeLib=static? Maybe they are wrapping it under mono library-mode and bypassing the BuildIntegration?
Also, for classification, we would need to consider <LinkerArg in user's csproj where this approach seems to be working.
There was a problem hiding this comment.
Why is static library reducing size compared to .o? Everything that we emit into .o is expected to be necessary.
There was a problem hiding this comment.
Archive .a is the static library format on Unix-like systems and a well-established standard for distributing compiled code. Whereas, .o is the intermediate form, object file. The linker knows how to read indexed symbol table out of .a for efficient linkage. With NativeLib=static, user expects and gets a static library.
There was a problem hiding this comment.
I do not think that anybody should be distributing these static libraries. It is too fragile system to be distributed and it does not behave like a regular static library (e.g. you cannot link two these to a project - they would collide).
This is only meant to be used locally and tightly integrated into some larger project build.
There was a problem hiding this comment.
Maybe we should stop calling this "static libraries" to avoid implying that the output behaves like a regular static library.
There was a problem hiding this comment.
e.g. you cannot link two these to a project - they would collide
That problem also exists with object files and the solution is the same; don't use multiple of them and put all code in a single compilation unit or pass -Wl,--allow-multiple-definition to the linker.
There was a problem hiding this comment.
This looks reasonable to me. I don't know if this should be enabled by default, it feels like we'd want to collect some feedback on this being used in real-world apps (that may have other native code with different requirements). I want us to have the option to change this without causing much uproar.
I think for .NET 9 we'll still keep static libraries as "not recommended", so maybe that lowers the bar on breaking changes by itself. Then it could be enabled by default still. Anyone have any thoughts?
|
We can set |
src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.targets
Outdated
Show resolved
Hide resolved
Fix #70277
This generates a markdown file with
-p:Nativelib=static. e.g. on osx-arm64:Test:
$ dotnet9 new classlib -n naotstatic && cd $_ $ dotnet9 publish -p:PublishAot=true -p:NativeLib=Static -o dist --ucr naotstatic.csproj | tee Determining projects to restore... All projects are up-to-date for restore. /Users/adeel/.dotnet9/sdk/9.0.100-preview.7.24321.3/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(326,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/Users/adeel/projects/naotstatic/naotstatic.csproj] naotstatic -> /Users/adeel/projects/naotstatic/bin/Release/net9.0/osx-arm64/naotstatic.dll naotstatic -> /Users/adeel/projects/naotstatic/dist/ + Linking instructions written to /Users/adeel/projects/naotstatic/dist/link-instructions.md.With
-p:SkipLinkingInstructions=true, it can be skipped.