Add app-local support for ICU

[safern]

Fixes: #826
This adds supports for apps to carry their own copy of ICU.

How this works

Apps need to set a runtime config property:

"System.Globalization.AppLocalIcu": "<suffix>:<version> or <version>"

It can also be via an env var in the form of DOTNET_SYSTEM_GLOBALIZATION_APPLOCALICU.

<suffix>: this is optional in the config property, but this follows the public ICU packaging conventions as when building a custom ICU you can customize it to produce the lib names and exported symbol names to contain a suffix. i.e: libicuucmyapp where myapp is the suffix. This can't be greater than 20 chars in length for the config switch.
<version>: this has to be a valid ICU version, i.e: 67.1. This version will be used to load the binaries and to get the exported symbols.

How do we load ICU

To load ICU we use NativeLibrary.TryLoad api which does probing in different paths, first it tries to find the library in NATIVE_DLL_SEARCH_DIRECTORIES property which is created by the dotnet host based on the deps.json file for the app. More explained here

For self contained apps, the user doesn't really need to do anything special but to make sure the app has ICU side by side in the APP directory, as for self contained apps, the app directory is by default in NATIVE_DLL_SEARCH_DIRECTORIES.

If you're consuming ICU via a NuGet package, this will work in framework-dependent apps as NuGet will resolve the native assets and include them in the deps.json file and in the output directory for the app under the runtimes dir, then we will load it from there.

However, the tricky part comes whenever it is a framework-dependent app (no self contained) and ICU is part of the project. The SDK doesn't yet have a feature for "loose" native binaries to land into deps.json: dotnet/sdk#11373. However, there is a workaround by adding something like this to the csproj:

  <ItemGroup>
    <IcuAssemblies Include="icu\*.so*" />
    <RuntimeTargetsCopyLocalItems Include="@(IcuAssemblies)" AssetType="native" CopyLocal="true" DestinationSubDirectory="runtimes/linux-x64/native/" DestinationSubPath="%(FileName)%(Extension)" RuntimeIdentifier="linux-x64" NuGetPackageId="System.Private.Runtime.UnicodeData" />
  </ItemGroup>

Note that this will have to be done for all the ICU binaries for the supported runtimes. Also, the NuGetPackageId metadata in the RuntimeTargetsCopyLocalItems item group, needs to match a NuGet package that the project actually references, it can't just be a dummy NuGet package.

macOS behavior

MacOS has a different behavior for resolving dependent dynamic libraries from the load commands specified in the match-o file than the Linux loader. In the Linux loader, we could just load libicudata first, then libicuuc and last libicui18n in that order to satisfy dependent libraries.

However, in MacOS this doesn't work. When building ICU in MacOS, you by default get a dynamic library with these load commands in libicuuc for example:

santifdezm@santiagos-mbp dlopen-test % otool -L /Users/santifdezm/repos/icu-build/icu/install/lib/libicuuc.67.1.dylib
/Users/santifdezm/repos/icu-build/icu/install/lib/libicuuc.67.1.dylib:
 libicuuc.67.dylib (compatibility version 67.0.0, current version 67.1.0)
 libicudata.67.dylib (compatibility version 67.0.0, current version 67.1.0)
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
 /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)

These commands are just referencing the name of the dependent libraries for the other components of ICU, so the loader will do the search following the dlopen conventions. Which involve having these libraries in the system directories or setting the LD_LIBRARY_PATH env vars, or having ICU at the app level directory.

However, there are some directives for the loader, like @loader_path which tells the loader to search for that dependency in the same directory as the binary with that load command. So there's 2 ways to achieve this.

instal_name_tool -change

Running:

install_name_tool -change "libicudata.67.dylib" "@loader_path/libicudata.67.dylib" /path/to/libicuuc.67.1.dylib
install_name_tool -change "libicudata.67.dylib" "@loader_path/libicudata.67.dylib" /path/to/libicui18n.67.1.dylib
install_name_tool -change "libicuuc.67.dylib" "@loader_path/libicuuc.67.dylib" /path/to/libicui18n.67.1.dylib

patching ICU to produce the install names with @loader_path

Changing: https://github.com/unicode-org/icu/blob/ef91cc3673d69a5e00407cda94f39fcda3131451/icu4c/source/config/mh-darwin#L32-L37 to:

LD_SONAME = -Wl,-compatibility_version -Wl,$(SO_TARGET_VERSION_MAJOR) -Wl,-current_version -Wl,$(SO_TARGET_VERSION) -install_name @loader_path/$(notdir $(MIDDLE_SO_TARGET))

before running autoconf (./runConfigureICU).

Homebrew does this for all the installed packages: https://github.com/Homebrew/brew/blob/d5ffb96d241e6a19d6ac4563f93210821db23245/Library/Homebrew/extend/os/mac/keg_relocate.rb#L110
https://github.com/Homebrew/brew/blob/ef471e97675b97a87827bda2542e39fc07bfe7b3/Library/Homebrew/os/mac/keg.rb#L24

Also, Firefox patches ICU similar, using the @executable_path directive for the loader:
https://hg.mozilla.org/mozilla-central/file/tip/intl/icu-patches/bug-915735#l20

cc: @ericstj @danmosemsft @tarekgh @jkotas @janvorli @stephentoub

FYI: @jefgen @mjsabby

[ghost]

Tagging subscribers to this area: @tarekgh, @safern, @krwq
Notify danmosemsft if you want to be subscribed.

[safern]

Forgot to mention. I need to add tests for this, but I'm working on building a test package in runtime-assets repo with ICU in it.

[safern]

cc: @GrabYourPitchforks

[mjsabby]

cc @timmydo @gregdunham

[safern]

I've addressed all PR feedback except for: #35383 (review) which I plan to address today. Could I please get more reviews in the meantime?

[safern]

All feedback is addressed 😄

[safern]

@tarekgh @jkotas @stephentoub does this look good now?

[safern]

Thanks all for the reviews 😄

[danmoseley]

yay!!!! 🔥