somehow get CI automated testing working for more platforms

[andrewrk]

Currently we have AppVeyor for Windows i386/x86_64 and Travis CI for MacOS x86_64 and Linux x86_64.

In order to support more targets in the standard library, we need more automatic testing.

• Linux i386
• FreeBSD x86_64
• FreeBSD ARM 32 bit
• FreeBSD ARM 64 bit
• Linux ARM 32 bit
• Linux ARM 64 bit
• OpenBSD x86_64

There are more, but these are some important ones.

[tiehuis]

I'm not aware of any hosted CI service which supports those architectures (especially the BSD's).

Beyond building your own, leveraging QEMU within Travis may be an option. Here is a pretty good example of this strategy which covers all the listed targets and may be worth exploring further:
https://github.com/rust-lang/libc/tree/master/ci

[andrewchambers]

I have experience doing automated openbsd installs and don't mind doing that one.

[andrewrk]

Thanks for volunteering! So we would have a way to automatically run tests on OpenBSD when we push to master branch?

[andrewchambers]

Sure, to start I can make a script to generate an openbsd vm disk image that is ready to go.

Later we can decide how to hook it up to a CI system. It may be necessary to avoid crazy slow nested VM's on travis.

[andrewchambers]

#941 works for openbsd, however we get timeouts. This approach will work for linux arm also, so is good for a reference point. ccache gives large enough speed boosts I think qemu is feasible, however it is definitely too slow for travis, especially when we add more targets.

[ghost]

https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/

- no timeouts

• macOS, linux, window
• free for OSS

so if they have no timeouts, the windows timeout issue would be gone and you could use those slightly wasteful linux virtualization things @andrewchambers described

I'm sorry to have to report back that I read wrong in the announcement, you only get unlimited CI minutes but not unlimited minutes per build, bummer, sorry for the noise/ confusion.

news from the news
https://www.reddit.com/r/programming/comments/9enz31/announcing_azure_pipelines_with_unlimited_cicd/e5sda7y/

you can specify up to 360min runtime in the yaml file

[jeremyepling]

I'm a product manager on Azure Pipelines. You can have multiple jobs in a single pipeline/yaml and each job has a limit of 6 hours. Do you need more time than that?

If you need a VM/agent that's different from what we support in the hosted pool, we have private agents for Linux, macOS, and Windows where you can use your own VMs. We're updating the open source offer to give 10 free private agents and unlimited minutes. Private agents have no time limit on individual jobs.

Ping me if you run into any issues.

[andrewrk]

Thanks @monouser7dig and @jeremyepling, I think this is worth trying out. 6 hours is plenty of time to run all the release tests as well as debug tests.

Would this let us have VMs with FreeBSD or OpenBSD on it? From a quick scan of the documentation, it looks like the way it works is to install Microsoft's agent code in the VM, but the agent only supports linux/windows/mac. Is that right?

With 6 hours to run tests, I would try the qemu setup that @andrewchambers did a proof-of-concept for, where we can add ARM support and other CPU architectures.

[jeremyepling]

@andrewrk, The Azure Pipelines Agent is based on .NET Core and it doesn't support FreeBSD or OpenBSD, but it might be coming soon.

[kristate]

@jeremyepling this is good news!

[andrewrk]

@jeremyepling Here is where I am stuck right now: https://developercommunity.visualstudio.com/content/problem/373572/unable-to-reference-secure-file.html

Also, is there a recommended way to cache? 90% of the time will be spent building LLVM 7 over and over again, and this could be cached. edit currently I am using s3 to cache.

[andrewrk]

Looks like the job only gets 60 minutes. How can I get the 6 hours?

[ghost]

Looks like the job only gets 60 minutes. How can I get the 6 hours?

@andrewrk
https://www.reddit.com/r/programming/comments/9enz31/announcing_azure_pipelines_with_unlimited_cicd/e5sda7y/

[jeremyepling]

There is a doc on timeouts. Set timeoutInMinutes to 0. That will give you the full 360 minutes.

jobs:
- job: Test
  timeoutInMinutes: 0

@andrewrk, Are you looking for caching across pipeline runs/build or across jobs within the same pipeline run/build?

[mgxm]

FreeBSD x86_64 should be ticked off 4840600

[dropwhile]

@andrewrk Not sure if this is interesting/useful information or not.....

I use buildkite at work, and it works pretty well. buildkite operates as a "head node" service, with the worker agent being installed and run "on your own systems". Likely best if those systems are dedicated to CI, because they may be running untrusted code in PRs.

buildkite apparently support several platforms, and appear to offer free open source project accounts. Apparently bazel uses it?

Might be an option if people and/or companies are willing to donate some VM hours to the project?

[andrewrk]

I think this is basically solved now that we have QEMU integrated into our test matrix + bootstrap repo. Each architecture/os combo has its own issues for getting to tier 1 support.