Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[windows] now can build android-toolchain.mdproj #743

Merged
merged 2 commits into from
Aug 23, 2017

Conversation

jonathanpeppers
Copy link
Member

@jonathanpeppers jonathanpeppers commented Aug 11, 2017

Problems fixed on Windows:

  • Many RequiredPrograms are n/a on Windows
  • Setup AndroidSdkItem for Windows downloads
  • AcceptAndroidSdkLicenses should run .bat file on Windows
  • UnzipDirectoryChildren was running /bin/mv and hitting
    PathTooLongException while using ZipFile.ExtractToDirectory
  • references to libzip.mdproj are conditional on Windows, will be
    getting libzip from NuGet in the future

UnzipDirectoryChildren:

  • on Windows we are using System.IO.Compression to extract
    directly into the destination and avoid %TEMP% to
    workaround MAX_PATH
  • Other platforms continue to use unzip, in order to preserve
    file attributes

General changes:

  • gitignore for VS 2017
  • Bumped LibZipSharp

You can see a successful build I setup on AppVeyor, which ran:

appveyor DownloadFile https://gist.githubusercontent.com/jonathanpeppers/50921eb6e839e1cb2f3975a01098f1a9/raw/e588b5222a7a06c1395614f5a2757d5adb4e4913/Configuration.OperatingSystem.props
git submodule update --init --recursive
nuget restore Xamarin.Android.sln
nuget restore external\LibZipSharp\libZipSharp.sln
msbuild build-tools\android-toolchain\android-toolchain.mdproj /v:detailed

@dnfclas
Copy link

dnfclas commented Aug 11, 2017

@jonathanpeppers,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

@jonathanpeppers
Copy link
Member Author

@jonpryor is there a way I could skip an entire MSBuild project on Windows? This PR skips a couple broken targets on Windows in libzip.mdproj, but it could probably skip the entire project when I change it to pull the dependency from NuGet.

@jonpryor
Copy link
Member

is there a way I could skip an entire MSBuild project on Windows?

Plausibly, yes, but only for .csproj-initiated builds, not .sln-initiated builds: You make the @(ProjectReference) conditional! That won't work for .sln builds, as that'll build all projects listed in the .sln regardless.

@@ -135,4 +154,7 @@
<HostOS></HostOS>
</AntItem>
</ItemGroup>
<ItemGroup>
<RequiredProgram Include="7z" Condition=" '$(HostOS)' == 'Windows' " />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be in build-tools/dependencies/dependencies.projitems.

@@ -16,7 +16,11 @@ public override bool Execute ()
var licdir = Path.Combine (Path.Combine (AndroidSdkDirectory, "licenses"));
Directory.CreateDirectory (licdir);

var psi = new ProcessStartInfo (Path.Combine (AndroidSdkDirectory, "tools", "bin", "sdkmanager"), "--licenses") { UseShellExecute = false, RedirectStandardInput = true };
var path = Path.Combine (AndroidSdkDirectory, "tools", "bin", "sdkmanager");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should use Which.GetProgramLocation(), so that %PATHEXT% is used. This way, if sdkmanager becomes sdkmanager.cmd in the future, we'll do the sane thing.

@jonathanpeppers jonathanpeppers force-pushed the windows-android-toolchain branch from 28cbd1a to 85e231a Compare August 11, 2017 20:17
@dellis1972
Copy link
Contributor

Tried using LibZipSharp, but could not get the native libraries to
load inside an MSBuild task

@jonathanpeppers this is probably down to you needing the libzip.dll in the root (next to the assembly) and in the x64 directory under it. We do use LibZipSharp in the android build tasks so it should work.

@jonathanpeppers
Copy link
Member Author

@dellis1972 I didn't try an x64 directory, I'll try again to see if I can get it to work.

I was trying something like this in BootstrapTasks.csproj which copied to bin\BuildDebug:

  <ItemGroup>
    <Content Include="..\..\external\LibZipSharp\packages\libzip.redist.1.1.2.7\build\native\bin\Win32\v140\Release\zip.dll" Condition=" '$(HostOS)' == 'Windows'">
      <Link>libzip.dll</Link>
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
    <Content Include="..\..\external\LibZipSharp\packages\grpc.dependencies.zlib.redist.1.2.8.10\build\native\bin\v140\Win32\Release\dynamic\stdcall\zlib.dll" Condition=" '$(HostOS)' == 'Windows'">
      <Link>zlib.dll</Link>
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
  </ItemGroup>

@jonathanpeppers
Copy link
Member Author

@dellis1972

needing the libzip.dll in the root (next to the assembly) and in the x64 directory under it.

Hmm, I feel like I should be able to get this to work, but I get System.DllNotFoundException: Unable to load DLL 'libzip.dll': The specified procedure could not be found..

Here is my directory after I added 64-bit dlls:

> ls -r .\bin\BuildDebug\

    Directory: ~\Desktop\Git\xamarin-android\bin\BuildDebug

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        8/11/2017   4:11 PM                x64
-a----         8/9/2017   2:51 PM          42496 api-merge.exe
-a----         8/9/2017   2:51 PM          97792 api-merge.pdb
-a----         8/9/2017   2:51 PM           5120 api-xml-adjuster.exe
-a----         8/9/2017   2:51 PM          11776 api-xml-adjuster.pdb
-a----         8/9/2017   2:51 PM         128512 jnienv-gen.exe
-a----         8/9/2017   2:51 PM          95744 jnienv-gen.pdb
-a----        4/19/2016   3:25 PM          68608 libzip.dll
-a----        8/11/2017   4:07 PM          90624 libZipSharp.dll
-a----        8/11/2017   4:07 PM         257536 libZipSharp.pdb
-a----         8/9/2017   2:51 PM           5632 remap-assembly-ref.exe
-a----         8/9/2017   2:51 PM          13824 remap-assembly-ref.pdb
-a----        8/11/2017   2:16 PM          40960 xa-prep-tasks.dll
-a----        8/11/2017   2:16 PM         101888 xa-prep-tasks.pdb
-a----         8/9/2017   2:51 PM         383488 Xamarin.Android.Cecil.dll
-a----         8/9/2017   2:51 PM        1340928 Xamarin.Android.Cecil.pdb
-a----         8/9/2017   2:51 PM          61440 Xamarin.Android.Tools.ApiXmlAdjuster.dll
-a----         8/9/2017   2:51 PM         165376 Xamarin.Android.Tools.ApiXmlAdjuster.pdb
-a----        8/11/2017   4:11 PM          34304 Xamarin.Android.Tools.BootstrapTasks.dll
-a----        8/11/2017   4:11 PM          83456 Xamarin.Android.Tools.BootstrapTasks.pdb
-a----        12/3/2015  10:22 AM          71168 zlib.dll

    Directory: ~\Desktop\Git\xamarin-android\bin\BuildDebug\x64

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        4/19/2016   3:25 PM          80896 libzip.dll
-a----        12/3/2015  10:22 AM          81920 zlib.dll

I printed out Environment.Is64BitProcess inside the task and it's printing False. We can talk on Monday.

@jonpryor
Copy link
Member

macOS+msbuild is failing:

error MSB4018: System.DllNotFoundException: libzip.dll [/Users/builder/jenkins/workspace/xamarin-android-msbuild-pr-builder/build-tools/bundle/bundle.mdproj]

@jonathanpeppers
Copy link
Member Author

@jonpryor I did mess something up on MacOS, I fixed in this PR: dotnet/android-libzipsharp#24

Is @grendello out this week?

We can wait to merge this until after our call tomorrow if you like.

@jonathanpeppers jonathanpeppers force-pushed the windows-android-toolchain branch 3 times, most recently from aca8d93 to a0873ba Compare August 17, 2017 16:00
@jonathanpeppers
Copy link
Member Author

@jonpryor I think this one is ready if you want to take another look. I think these test failures are something else.

@jonpryor
Copy link
Member

@jonathanpeppers: Those unit tests -- the AOT and mkbundle-related tests -- failed because -- for some reason -- the prebuilt bundle-*.zip file wasn't found:

Downloading `https://xamjenkinsartifact.blob.core.windows.net/xamarin-android/xamarin-android/bin/Debug/bundle-v19-he28883ec-Debug-Darwin-libzip=1d8b1ac,llvm=dbb6fdf,mono=143421c.zip` to `../../bin/BuildDebug/.bundle-v19-he28883ec-Debug-Darwin-libzip=1d8b1ac,llvm=dbb6fdf,mono=143421c.zip.download`.
warning : Unable to download URL `https://xamjenkinsartifact.blob.core.windows.net/xamarin-android/xamarin-android/bin/Debug/bundle-v19-he28883ec-Debug-Darwin-libzip=1d8b1ac,llvm=dbb6fdf,mono=143421c.zip` to `../../bin/BuildDebug/bundle-v19-he28883ec-Debug-Darwin-libzip=1d8b1ac,llvm=dbb6fdf,mono=143421c.zip`: 404 (The specified blob does not exist.)

Without a pre-existing bundle, the AOT compilers will only be built if:

  1. This is the xamarin-android job, not the PR builder, or
  2. The PR has the full-mono-integration-build label.

Neither of which is true for this PR.

If this PR had a mono bump, or had a bundle version bump (in bundle-path.targets), or one of the @(VersionFile) files within bundle-path.targets had changed in the commit, that would make sense.

If there was some bizarre network error, and the bundle Url was a 404 for the PR build, but otherwise exists, that would also make sense.

Unfortunately, neither of the above is true:

$ curl -o b.zip 'https://xamjenkinsartifact.blob.core.windows.net/xamarin-android/xamarin-android/bin/Debug/bundle-v19-he28883ec-Debug-Darwin-libzip=1d8b1ac,llvm=dbb6fdf,mono=143421c.zip` to `../../bin/BuildDebug/bundle-v19-he28883ec-Debug-Darwin-libzip=1d8b1ac,llvm=dbb6fdf,mono=143421c.zip'
$ cat b.zip 
<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.
RequestId:fdc8a84d-0001-00b9-1bbc-172b41000000
Time:2017-08-18T00:54:11.3852133Z</Message></Error>

That is worrying. :-(

The Ubuntu+xbuild and macOS+msbuild PR builds don't fail because they don't run the unit tests, thus the lack of AOT compilers won't be a problem.

@jonpryor
Copy link
Member

I don't understand why the hash value is changing. My current ~master build is using bundle-v19-h2a66f94e-Debug-Darwin-libzip=1d8b1ac,llvm=dbb6fdf,mono=143421c.zip. Compare to what this PR is downloading:

bundle-v19-h2a66f94e-Debug-Darwin-libzip=1d8b1ac,llvm=dbb6fdf,mono=143421c.zip
bundle-v19-he28883ec-Debug-Darwin-libzip=1d8b1ac,llvm=dbb6fdf,mono=143421c.zip

Scroll back to the <HashFileContents/> invocation, and:

Task "HashFileContents"
        Using task HashFileContents from Xamarin.Android.BuildTools.PrepTasks.HashFileContents, xa-prep-tasks, Version=1.0.6438.28829, Culture=neutral, PublicKeyToken=null
        Task HashFileContents
          AbbreviatedHashLength: 8
          HashAlgorithm: SHA1
          Files:
            /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/mono-runtimes/mono-runtimes.mdproj
            /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/mono-runtimes/mono-runtimes.projitems
            /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/mono-runtimes/mono-runtimes.props
            /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/mono-runtimes/mono-runtimes.targets
            ../scripts/BuildEverything.mk
          [Output] AbbreviatedCompleteHash: e28883ec
          [Output] CompleteHash: e28883ec702b1bef48eea431aacf0295fb3ecdac
          [Output] Hashes:
            /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/mono-runtimes/mono-runtimes.mdproj: a2d5acfc8969eed2ffad38a60703f946df26f5a2:
            /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/mono-runtimes/mono-runtimes.projitems: 51ec244cc5e5524a657aa97d6016cdb7c75c6624:
            /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/mono-runtimes/mono-runtimes.props: 0798cd7c7347f037121ccd5b074430595179a37c:
            /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/mono-runtimes/mono-runtimes.targets: 83a47249c3fd68dc11fdac288d1501701db17d6b:
            /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/scripts/BuildEverything.mk: c226a81a6436bd10ed6ec3b948c6538ebcacad11:

Compare to my local files:

$ cat build-tools/mono-runtimes/mono-runtimes.mdproj | tr -d '\r\n' | shasum
a2d5acfc8969eed2ffad38a60703f946df26f5a2  -
$ cat build-tools/mono-runtimes/mono-runtimes.projitems | tr -d '\r\n' | shasum
51ec244cc5e5524a657aa97d6016cdb7c75c6624  -
$ cat build-tools/mono-runtimes/mono-runtimes.props | tr -d '\r\n' | shasum
0798cd7c7347f037121ccd5b074430595179a37c  -
$ cat build-tools/mono-runtimes/mono-runtimes.targets | tr -d '\r\n' | shasum
83a47249c3fd68dc11fdac288d1501701db17d6b  -
$ cat build-tools/scripts/BuildEverything.mk | tr -d '\r\n' | shasum
d9c6bed2a6cc3f9be68944002e9c094a99e74ab2  -

BuildEverything.mk is the only file where things don't match: c226a81a6436bd10ed6ec3b948c6538ebcacad11 (PR) != d9c6bed2a6cc3f9be68944002e9c094a99e74ab2 (commit 14f1bb5).

That's very confusing, because BuildEverything.mk isn't listed as a file changed in this PR.

@jonpryor
Copy link
Member

build

@jonpryor
Copy link
Member

I forgot about commit d891a86, which explains the change to BuildEverything.mk. Sanity restored!

@@ -5,15 +5,15 @@
<_AptGetInstall>apt-get -f -u install</_AptGetInstall>
</PropertyGroup>
<ItemGroup>
<RequiredProgram Include="$(HostCcName)" />
<RequiredProgram Include="$(HostCxxName)" />
<RequiredProgram Include="$(HostCcName)" Condition=" '$(HostOS)' != 'Windows' " />
Copy link
Member

@jonpryor jonpryor Aug 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's talk about "beauty": I think code should be pretty.

"Pretty" is not well defined, but to me, it's this: things should be horizontally aligned whenever practical.

Specifically, these Condition attributes: note that every other Condition attribute in this file is horizontally aligned.

Would you mind fixing the indentation here and elsewhere in this file? :-)

@jonathanpeppers jonathanpeppers force-pushed the windows-android-toolchain branch from a0873ba to e9bc2a6 Compare August 18, 2017 02:17
@jonathanpeppers
Copy link
Member Author

Hmm, this one downloaded the bundle and those AOT tests passed this time, but some others failed:

Xamarin.Android.Build.Tests.BuildTest.BuildMultiDexApplication(False,"v7.1")
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v7.1","24.0.1","1.8.0_101",True)
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v6.0","24.0.1","1.8.0_101",True)
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v6.0","24.0.0","1.7.0_101",True)
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v5.0","24.0.1","1.8.0_101",True)
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v5.0","24.0.0","1.7.0_101",True)
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v5.0","24.0.0","1.6.0_101",True)
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v7.1","24.0.1","1.6.x_101",True)

The previous build failure with the missing bundle had these failing:

Xamarin.Android.Build.Tests.BuildTest.BuildAotApplication("armeabi-v7a",False,True)
Xamarin.Android.Build.Tests.BuildTest.BuildAotApplication("armeabi-v7a",True,True)
Xamarin.Android.Build.Tests.BuildTest.BuildAotApplication("x86",False,True)
Xamarin.Android.Build.Tests.BuildTest.BuildAotApplication("x86",True,True)
Xamarin.Android.Build.Tests.BuildTest.BuildAotApplicationAndBundle("armeabi-v7a",False,True)
Xamarin.Android.Build.Tests.BuildTest.BuildAotApplicationAndBundle("armeabi-v7a",True,True)
Xamarin.Android.Build.Tests.BuildTest.BuildAotApplicationAndBundle("x86",False,True)
Xamarin.Android.Build.Tests.BuildTest.BuildAotApplicationAndBundle("x86",True,True)
Xamarin.Android.Build.Tests.BuildTest.BuildMultiDexApplication(False,"v7.1")
Xamarin.Android.Build.Tests.BuildTest.CheckSequencePointGeneration(True,True,True,True,"PdbOnly",False,"release")
Xamarin.Android.Build.Tests.BuildTest.CheckSequencePointGeneration(True,False,True,False,"",False,"release")
Xamarin.Android.Build.Tests.BuildTest.CheckSequencePointGeneration(True,True,True,False,"",False,"release")
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v7.1","24.0.1","1.8.0_101",True)
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v6.0","24.0.1","1.8.0_101",True)
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v6.0","24.0.0","1.7.0_101",True)
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v5.0","24.0.1","1.8.0_101",True)
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v5.0","24.0.0","1.7.0_101",True)
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v5.0","24.0.0","1.6.0_101",True)
Xamarin.Android.Build.Tests.BuildTest.ValidateJavaVersion("v7.1","24.0.1","1.6.x_101",True)

Problems fixed on Windows:
- Many `RequiredPrograms` are n/a on Windows
- Setup `AndroidSdkItem` for Windows downloads
- `AcceptAndroidSdkLicenses` should run `.bat` file on Windows
- `UnzipDirectoryChildren` was running `/bin/mv` and hitting
`PathTooLongException` while using `ZipFile.ExtractToDirectory`
- references to `libzip.mdproj` are conditional on Windows, will be
getting `libzip` from NuGet in the future

UnzipDirectoryChildren:
- on Windows we are using `System.IO.Compression` to extract
directly into the destination and avoid `%TEMP%` to
workaround `MAX_PATH`
- Other platforms continue to use `unzip`, in order to preserve
file attributes

General changes:
- gitignore for VS 2017
@jonathanpeppers jonathanpeppers force-pushed the windows-android-toolchain branch from e9bc2a6 to d1b8554 Compare August 21, 2017 19:39
@jonpryor
Copy link
Member

The macOS+xbuild PR Build job hung. I aborted it.

20:58:15 		adb I 08-21 20:58:15 40362 15889678 adb_io.cpp:75] readx: fd=3 wanted=4
# aborted...
23:11:32 		Tool /Users/builder/android-toolchain/sdk/platform-tools/adb execution finished.

@jonpryor jonpryor merged commit 467f42d into dotnet:master Aug 23, 2017
@jonathanpeppers jonathanpeppers deleted the windows-android-toolchain branch August 23, 2017 19:26
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Oct 29, 2020
jonpryor added a commit that referenced this pull request Oct 30, 2020
@github-actions github-actions bot locked and limited conversation to collaborators Feb 5, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants