Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tizen] CI for pull requests #2394

Closed
alpencolt opened this issue Jan 30, 2020 · 18 comments
Closed

[Tizen] CI for pull requests #2394

alpencolt opened this issue Jan 30, 2020 · 18 comments
Assignees
Milestone

Comments

@alpencolt
Copy link

Would be good to have CI builds for Tizen/armel to prevent merging PRs which breaks this platform.

cc @jkotas @gbalykov

@jkotas
Copy link
Member

jkotas commented Jan 30, 2020

Ok, here are the steps to make this happen:

We should start with builds only (no tests for now). We can look at tests later if you see too many functional breakages sneaking through.

@safern
Copy link
Member

safern commented Jan 30, 2020

cc @dotnet/runtime-infrastructure

@franksinankaya
Copy link
Contributor

Can we get some instructions?

@safern
Copy link
Member

safern commented Mar 10, 2020

@franksinankaya what kind of instructions do you need?

I would suggest starting with: Add sources for docker image with the right Tizen ROOTFS to https://github.com/dotnet/dotnet-buildtools-prereqs-docker repo

That repo should have all the necessary instructions, @MichaelSimons is the expert and admin of that repo, in case you need to reach out on the PR to add the necessary images.

Once you have dotnet/runtime building for armel locally with that generated image and the image is published on that repo, I can help setting up CI using that docker image.

@franksinankaya
Copy link
Contributor

franksinankaya commented Mar 10, 2020

@safern
Apologies for hijacking this thread. I'll move to "Need GCC build pipeline" issue.

@ViktorHofer ViktorHofer added this to the Future milestone Jul 15, 2020
@jkotas jkotas mentioned this issue Jul 7, 2021
@clamp03
Copy link
Member

clamp03 commented Jul 8, 2021

@alpencolt Do you have any progress on this issue? If you cannot handle it, could I take this issue?

@clamp03
Copy link
Member

clamp03 commented Jul 12, 2021

@alpencolt @gbalykov ARMEL build is broken again by #54954.

  [  2%] Building C object Native.Unix/System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native-Static.dir/home/dheon/work/dotnet/jobs/fnv/runtime/src/libraries/Native/AnyOS/brotli/common/platform.c.o
  /home/dheon/work/dotnet/jobs/fnv/runtime/src/coreclr/minipal/Unix/doublemapping.cpp:18:10: fatal error: 'linux/memfd.h' file not found 
  #include <linux/memfd.h>                                                                                                              
           ^~~~~~~~~~~~~~~                                                                                                                                                                                                                                                      
  1 error generated.

Could you add the armel + Tizen CI build leg? Or could I take this issue?
Thank you.

@alpencolt
Copy link
Author

@clamp03 unfortunately we had no time to this issue. Would be good if you'll take it.

@clamp03
Copy link
Member

clamp03 commented Jul 12, 2021

@alpencolt Thank you. I will handle it.

@clamp03
Copy link
Member

clamp03 commented Jul 13, 2021

I resolved. It is because an invisible character is added in windows GIT.

@safern Hi, could you help how to test https://github.com/dotnet/dotnet-buildtools-prereqs-docker locally?
I tested with .\build.ps1 -DockerfilePath "*ubuntu/18.04/*" command on Windows10 PC. (I don't change anything.).
It shows the below error message during the pre-build step of cross arm64.

-- EXECUTION ELAPSED TIME: 00:17:29.5494734
-- EXECUTING: /repo/src/ubuntu/18.04/cross/arm64/16.04/hooks/pre-build
env: can't execute 'sh
': No such file or directory
-- EXECUTION ELAPSED TIME: 00:00:00.0021056
Unhandled exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.InvalidOperationException: Failed to execute build hook '/repo/src/ubuntu/18.04/cross/arm64/16.04/hooks/pre-build'

env: can't execute 'sh
': No such file or directory
   at Microsoft.DotNet.ImageBuilder.ExecuteHelper.Execute(ProcessStartInfo info, Func`2 executor, Boolean isDryRun, String errorMessage, String executeMessageOverride) in /image-builder/src/ExecuteHelper.cs:line 75
   at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.InvokeBuildHook(String hookName, String buildContextPath) in /image-builder/src/Commands/BuildCommand.cs:line 594
   at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.BuildImage(PlatformInfo platform, IEnumerable`1 allTags) in /image-builder/src/Commands/BuildCommand.cs:line 369
   at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.BuildImages() in /image-builder/src/Commands/BuildCommand.cs:line 256
   at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.<ExecuteAsync>b__10_0() in /image-builder/src/Commands/BuildCommand.cs:line 58
   at Microsoft.DotNet.ImageBuilder.DockerHelper.ExecuteWithUser(Action action, String username, String password, String server, Boolean isDryRun) in /image-builder/src/DockerHelper.cs:line 35
   at Microsoft.DotNet.ImageBuilder.Commands.DockerRegistryCommand`2.ExecuteWithUser(Action action) in /image-builder/src/Commands/DockerRegistryCommand.cs:line 17
   at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.ExecuteAsync() in /image-builder/src/Commands/BuildCommand.cs:line 54
   at Microsoft.DotNet.ImageBuilder.Commands.Command`2.<GetCliCommand>b__9_0(TOptions options) in /image-builder/src/Commands/Command.TOptions.cs:line 46
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at System.Delegate.DynamicInvoke(Object[] args)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseErrorReporting>b__21_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass25_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__22_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseDirective>b__20_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseDebugDirective>b__11_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<<UseExceptionHandler>b__0>d.MoveNext()
Executing: 'docker container rm -f ImageBuilder-20210713112430'
ImageBuilder-20210713112430

Thank you.

@safern
Copy link
Member

safern commented Jul 14, 2021

@clamp03 did you figure this out? If not @michaelgsharp or @mthalman would know more about buildtools-prereqs-docker issues. I just see that it can't find sh in your environment, so I don't know if there is a special setup needed in order to build ubuntu images in Windows.

@mthalman
Copy link
Member

Cross-build Dockerfiles won't be buildable on a Windows machine because they'll run the Linux-based pre-build hook script which runs on the local machine. Is there a particular set of Dockerfiles you're attempting to test that doesn't include cross-build Dockerfiles?

@clamp03
Copy link
Member

clamp03 commented Jul 15, 2021

@safern @mthalman Thank you for the comments.
Could you please give a guide to test Cross-build Dockerfiles on non-Windows machine?

I made a draft for armel-tizen dotnet/dotnet-buildtools-prereqs-docker#478
On Windows 10 machine, it builds a docker image with /crossrootfs. With the docker image, I can build armel tizen runtime (cross build).
Could you please review it?

My concerns of the PR are

  • It builds a docker image with crossrootfs. However, it fails to run post-build hook. (Other cross builds like arm and arm64 fail to run post-build hook too on my environment.)
  • armel tizen rootfs doesn't contain /crossrootfs/$arch/bin, it fails in directory existence check. I updated to /crossrootfs/$arch/usr/bin. I checked armel, but I didn't validate the others because of my build environment.

@clamp03
Copy link
Member

clamp03 commented Jul 20, 2021

Ok, here are the steps to make this happen:

We should start with builds only (no tests for now). We can look at tests later if you see too many functional breakages sneaking through.

The first step is done. dotnet/dotnet-buildtools-prereqs-docker#478

@clamp03
Copy link
Member

clamp03 commented Jul 20, 2021

@safern Could you help me to add armel checked CI?

This is my draft. Is this the right approach?

diff --git a/eng/pipelines/common/platform-matrix.yml b/eng/pipelines/common/platform-matrix.yml
index defd34be84a..c75c00d5153 100644
--- a/eng/pipelines/common/platform-matrix.yml
+++ b/eng/pipelines/common/platform-matrix.yml
@@ -624,6 +624,32 @@ jobs:
         helixQueueGroup: ${{ parameters.helixQueueGroup }}
         ${{ insert }}: ${{ parameters.jobParameters }}
 
+# Tizen armel
+
+- ${{ if or(containsValue(parameters.platforms, 'Tizen_armel') }}:
+  - template: xplat-setup.yml
+    parameters:
+      jobTemplate: ${{ parameters.jobTemplate }}
+      helixQueuesTemplate: ${{ parameters.helixQueuesTemplate }}
+      variables: ${{ parameters.variables }}
+      osGroup: Linux
+      archType: armel
+      targetRid: tizen-armel
+      platform: Tizen_armel
+      container:
+        image: ubuntu-18.04-cross-armel-tizen-20210719212651-8b02f56
+        registry: mcr
+      jobParameters:
+        runtimeFlavor: ${{ parameters.runtimeFlavor }}
+        stagedBuild: ${{ parameters.stagedBuild }}
+        buildConfig: ${{ parameters.buildConfig }}
+        ${{ if eq(parameters.passPlatforms, true) }}:
+          platforms: ${{ parameters.platforms }}
+        helixQueueGroup: ${{ parameters.helixQueueGroup }}
+        crossBuild: true
+        crossrootfsDir: '/crossrootfs/armel'
+        ${{ insert }}: ${{ parameters.jobParameters }}
+
 # Windows x64
diff --git a/eng/pipelines/runtime.yml b/eng/pipelines/runtime.yml
index fdeeecdaa82..51e140ba082 100644
--- a/eng/pipelines/runtime.yml
+++ b/eng/pipelines/runtime.yml
@@ -72,6 +72,7 @@ jobs:
     - Linux_musl_arm64
     - Linux_musl_x64
     - OSX_arm64
+    - Tizen_armel
     - windows_x86
     - windows_x64
     - windows_arm
diff --git a/eng/pipelines/coreclr/templates/build-job.yml b/eng/pipelines/coreclr/templates/build-job.yml
index 699983fa0c3..d113bb7bdf2 100644
--- a/eng/pipelines/coreclr/templates/build-job.yml
+++ b/eng/pipelines/coreclr/templates/build-job.yml
@@ -141,6 +141,12 @@ jobs:
     - name: SignType
       value: $[ coalesce(variables.OfficialSignType, 'real') ]
 
+    - name: clrPortableBuildArg
+      value: ''
+    - ${{ if eq(parameters.archType, 'armel' )}}:
+      - name: clrPortableBuildArg
+        value: '--portablebuild false'
+
     - ${{ parameters.variables }}
 
     steps:
@@ -188,7 +194,7 @@ jobs:
 
     # Build CoreCLR Runtime
     - ${{ if ne(parameters.osGroup, 'windows') }}:
-      - script: $(Build.SourcesDirectory)/src/coreclr/build-runtime$(scriptExt) $(buildConfig) $(archType) $(crossArg) $(osArg) -ci $(compilerArg) $(clrBuildPALTestsBuildArg) $(pgoInstrumentArg) $(officialBuildIdArg) $(clrInterpreterBuildArg)
+      - script: $(Build.SourcesDirectory)/src/coreclr/build-runtime$(scriptExt) $(buildConfig) $(archType) $(crossArg) $(osArg) -ci $(compilerArg) $(clrBuildPALTestsBuildArg) $(pgoInstrumentArg) $(officialBuildIdArg) $(clrInterpreterBuildArg) $(clrPortableBuildArg)
         displayName: Build CoreCLR Runtime
     - ${{ if eq(parameters.osGroup, 'windows') }}:
       - script: set __TestIntermediateDir=int&&$(Build.SourcesDirectory)/src/coreclr/build-runtime$(scriptExt) $(buildConfig) $(archType) -ci $(enforcePgoArg) $(pgoInstrumentArg) $(officialBuildIdArg) $(clrInterpreterBuildArg)
@@ -201,7 +207,7 @@ jobs:
         displayName: Disk Usage after Build
 
     # Build CoreCLR Managed Components
-    - script: $(Build.SourcesDirectory)$(dir)build$(scriptExt) -subset clr.corelib+clr.nativecorelib+clr.tools+clr.packages+clr.paltestlist $(crossArg) -arch $(archType) $(osArg) -c $(buildConfig) $(pgoInstrumentArg) $(officialBuildIdArg) -ci
+    - script: $(Build.SourcesDirectory)$(dir)build$(scriptExt) -subset clr.corelib+clr.nativecorelib+clr.tools+clr.packages+clr.paltestlist $(crossArg) -arch $(archType) $(osArg) -c $(buildConfig) $(pgoInstrumentArg) $(officialBuildIdArg) $(clrPortableBuildArg) -ci
       displayName: Build managed product components and packages
 
     # Run CoreCLR Tools unit tests

@safern
Copy link
Member

safern commented Jul 22, 2021

@clamp03 that looks like you are going on the right direction. Also if you shared a branch it would be easier to diff, but overall looks like you are on the right path.

@clamp03
Copy link
Member

clamp03 commented Jul 26, 2021

@safern Thank you for your comment. I made a #56281 PR

@clamp03
Copy link
Member

clamp03 commented Oct 14, 2021

@alpencolt #56281 is merged. Please close this issue.

@safern safern closed this as completed Oct 14, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Nov 13, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

8 participants