[ios, macos, build] Add CircleCI iOS & macOS builds

[friedbunny]

Follows up on #8551 & #9051 and adds iOS & macOS tests to CircleCI. A new branch takes about 8-9 minutes to finish running these tests, while a cached build can take 4-5 minutes. This does not yet replace Bitrise and these builds will not be marked required, but that’s the end goal after some period of validation.

Adds three four builds, currently all using Xcode 9.0.x and iOS 11.x/macOS 10.13.x:

• ios-debug: Debug build that runs unit tests on a single iPhone 7 simulator. Replaces the Bitrise iOS build.
• ios-sanitize-address: Same, but with the address sanitizer enabled. Fairly slow and currently times out a few tests — will probably disable this, disabled this until tests can be adjusted.
• ios-sanitize-thread: Same, but with the thread sanitizer enabled. Seems about as fast/reliable as without.
• macos-debug: Debug build that runs core and macOS unit tests. Replaces the Bitrise macOS build.

You can see these builds for this branch here: https://circleci.com/gh/mapbox/mapbox-gl-native/tree/fb-circleci-agua

Selected benefits

• Faster
• Auto-cancels obsolete builds(?)
• Can cancel groups of builds

Notes

• CircleCI 2.0’s macOS support is early and I’m seeing containers fail to start (Unexpected preparation error: Process exited with status 1) and then automatically retry. This initially appears as a failure, but then later reports its actual status when the retry has completed.
• Xcode 9 adds the ability to run multiple concurrent simulators, but this seems very unreliable and tests randomly fail. Xcpretty/scan also don’t handle this well, failing to echo the test output from any of the simulators. Too bad, because multiple concurrent simulators would be awesome.
• After we’re in a good place, I’ll work on porting workflows that require auth or signing, such as the deployment flow or the nightly builds.
• Qt and Node platform builds that we’re running on macOS will also come, eventually.

Questions

• What types of builds should we have? Release? Older iOS simulators? Older Xcode? Linting?

/cc @mapbox/ios @kkaefer @brunoabinader

[kkaefer]

This is the same path as above?

[friedbunny]

The build and test steps are separated, but output logs to the same path with the same name. There’s not a mechanism to have the files named differently (and the second would overwrite the first), so they’re copied from the same source folder to differently named artifacts folders.

[friedbunny]

I agree that this is inelegant — maybe having a step that renames the logs would be better.

[brunoabinader]

I guess we could also export CCACHE_DIR=/root/.ccache in the iOS CI jobs instead of adding this.

[friedbunny]

Thanks, I’ll look into that. /root may not exist or may have permissions issues on macOS... 🤔

[friedbunny]

I tried this and:

• We’d have to export the cache path for every macOS-based job — it’s not enough to do this in a step, it has to be global for the job.
• We’d have to create the /root/.ccache directory with sudo and give it proper permissions (or do something similar with a symlink).

I think we should just leave it as-is, saving/restoring both possible ccache paths. Caches aren’t shared across platforms and a non-existent path doesn’t break the save/restore steps.

[jfirebaugh]

This looks great!

[friedbunny]

I added a macOS build that runs the core and platform unit tests — cold build is ~10 minutes, cached is ~5 minutes.

I don’t have immediate plans to port the Node and Qt builds from Bitrise (as I’m not very familiar with them), but maybe that’s something @brunoabinader would like to look into doing? 🙇 Those are the last two remaining regular gl-native builds that run on Bitrise, after this PR.

[friedbunny]

I think this is ready to merge.

• Squashed and rebased on latest release-aqua.
• Disabled the iOS and macOS workflows on Bitrise in 9deb1a39533abc5e0643f6dacfb9831c2ac9c3bd (by making them do nothing — they will always report as passing).
• Once this is merged, we can:
  • Make the new CircleCI builds required for release-aqua. (I went ahead and did this already; it shouldn’t affect branches that don’t know about the new CircleCI builds.)
  • Un-require the macOS and iOS Bitrise builds for release-agua.
  • Cherry-pick these changes into master and change required builds there, too → [build] Add CircleCI iOS & macOS builds (master branch) #10289.

We will need to keep Bitrise available in this repo until all branches no longer need it and the remaining platforms and nightly builds have been migrated.

[brunoabinader]

Ticket for CircleCI Node & Qt macOS builds 👉 #10278

[friedbunny]

The three new CircleCI builds — ios-debug, ios-sanitize-thread, and macos-debug are now required on the release-agua branch.

If you find that your PR fails to start these tests: rebase onto release-agua to bring in these changes.

/cc @mapbox/android @mapbox/gl-core

[friedbunny]

Failures you might encounter

Unexpected preparation error

CircleCI 2.0 macOS seems to have some issues starting up builds. You may see a build fail and auto-retry multiple times before it starts installing our stuff.

GitHub and CircleCI may report a build in this state as “failed”, but if you allow it to run and it passes, the state will be updated appropriately.

Unexpected preparation error: Process exited with status 1

There was an issue while running this container and it was rerun. The most recent run is shown.

... or the bigger sibling:

Build-agent version 0.0.4457-19d0729 (2017-10-26T08:06:09+0000)
Creating a dedicated VM with xcode:9.0 image
Assigned VM request id: 2572019
  waiting .......
We had an unexpected error preparing a VM for this build, potentially due to our infrastructure or cloud provider.  Please retry the build in a few minutes
Unexpected environment preparation error: VM creation failed

Remediation: let it auto-retry and build (even if it says “failed”); rebuild if it appears stalled.

Failed to download dependencies

You may see dependencies explicitly fail to download or, worse, silently fail to download and then be missing later.

CMake Error at cmake/mason.cmake:103 (message):
  [Mason] Failed to download
  https://mason-binaries.s3.amazonaws.com/ios-8.0/icu/58.1-min-size.tar.gz:
  curl: (56) SSLRead() return error -9845

#!/bin/bash -eo pipefail
brew install cmake
brew install ccache

^D��^D��Error: Operation already in progress for cmake.formula
Another active Homebrew process is already using cmake.formula.
Please wait for it to finish or terminate it to continue.
Exited with code 1

node platform/darwin/scripts/check-public-symbols.js macOS iOS
module.js:529
    throw err;
    ^

Error: Cannot find module 'lodash'

Remediation: rebuild.

[friedbunny]

Heard from CircleCI support that “Unexpected preparation error” failures should be fixed — LMK if anyone continues to see them.