Add Google Cloud SDK #36

bootstraponline · 2016-11-21T17:09:09Z

Fix https://github.com/bitrise-docker/android/issues/35

viktorbenei · 2016-11-21T17:12:26Z

Thanks for the PR!

A note: please don't change the HOME env!
Also, see the related GH issue for an alternative way of using the google cloud SDK, using the official google cloud sdk docker image and docker run: https://github.com/bitrise-docker/android/issues/35#issuecomment-262001948

viktorbenei · 2016-11-21T17:12:49Z

Dockerfile

+# --- https://firebase.google.com/docs/test-lab/command-line
+
+# Install the Google Cloud SDK.
+ENV HOME /


please don't change the HOME env

viktorbenei · 2016-11-21T17:15:03Z

Dockerfile

+# Install the Google Cloud SDK.
+ENV HOME /
+ENV CLOUDSDK_PYTHON_SITEPACKAGES 1
+RUN wget https://dl.google.com/dl/cloudsdk/channels/rapid/google-cloud-sdk.zip && unzip google-cloud-sdk.zip && rm google-cloud-sdk.zip


the SDK is available as an official apt package, it's usually better to use that. See the apt-get (Debian and Ubuntu only) section at https://cloud.google.com/sdk/downloads?hl=sr#linux

Apt provides no control over the configuration, for example installing only the beta components. These steps are from the official Docker image (where they also don't install via apt).

I will test to see if the apt install contains the necessary components.

Apt provides no control over the configuration, for example installing only the beta components.

AFAIK the apt versions include every component - or do you mean that there are components which are not available in the APT version?

AFAIK the apt versions include every component

No, certain components are broken out into dedicated packages.

Optionally install any of these additional components: google-cloud-sdk-app-engine-python google-cloud-sdk-app-engine-java google-cloud-sdk-datastore-emulator google-cloud-sdk-pubsub-emulator google-cloud-sdk-bigtable-emulator kubectl

I don't see one for beta which is required for testing. I'll know for sure in a bit once I finish exploring the package.

Update: beta is installed by default.
https://github.com/bitrise-docker/android/issues/35#issuecomment-262027376

viktorbenei · 2016-11-21T18:07:28Z

Error for the first build:

Removing intermediate container 230d2b629f97
Step 62 : RUN sed -i -- 's/\"disable_updater\": false/\"disable_updater\": true/g' /google-cloud-sdk/lib/googlecloudsdk/core/config.json
 ---> Running in 513420058e70
sed: can't read /google-cloud-sdk/lib/googlecloudsdk/core/config.json: No such file or directory
The command '/bin/sh -c sed -i -- 's/\"disable_updater\": false/\"disable_updater\": true/g' /google-cloud-sdk/lib/googlecloudsdk/core/config.json' returned a non-zero code: 2

bootstraponline · 2016-11-21T18:08:25Z

I'm seeing the errors also. I'll post an update once I've confirmed this is working.

viktorbenei · 2016-11-21T19:02:29Z

Dockerfile

+ && sudo apt-get install -y -qq google-cloud-sdk
+
+RUN /usr/bin/gcloud config set --installation component_manager/disable_update_check true
+RUN sed -i -- 's/\"disable_updater\": false/\"disable_updater\": true/g' /usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/config.json


is this required even if you run gcloud config set --installation component_manager/disable_update_check true ?

Isn't gcloud config set --installation component_manager/disable_update_check true enough?

No, see the upstream comment:

# Disable updater completely. # Running `gcloud components update` doesn't really do anything in a union FS. # Changes are lost on a subsequent run. RUN sed -i -- 's/\"disable_updater\": false/\"disable_updater\": true/g' /google-cloud-sdk/lib/googlecloudsdk/core/config.json

I also manually confirmed that gcloud config set never updates the config.json file.

Makes sense, thank you!

viktorbenei · 2016-11-21T19:03:56Z

Dockerfile

+# --- https://firebase.google.com/docs/test-lab/command-line
+# --- https://cloud.google.com/sdk/downloads?hl=sr#linux
+
+ENV CLOUD_SDK_REPO="cloud-sdk-xenial"


CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" would be better, without that we'll have to update this variable from xenial when we update the base Ubuntu version

Back ticks should work, we use it here for example: https://github.com/bitrise-docker/bitrise-base/blob/master/Dockerfile#L95

E: Failed to fetch https://packages.cloud.google.com/apt/dists/cloud-sdk-$(lsb_release/-c/binary-amd64/Packages 404 Not Found

bootstraponline · 2016-11-21T19:04:23Z

Ok, this is ready for review.

docker-compose build completes successfully.
gcloud version lists all the relevant version info for the system report.

viktorbenei · 2016-11-21T19:07:06Z

Dockerfile

+
+ENV CLOUD_SDK_REPO="cloud-sdk-xenial"
+
+RUN echo "deb https://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" |  sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list \


I think, as $CLOUD_SDK_REPO is not used anywhere else, we could remove the ENV line and instead include

cloud-sdk-`lsb_release -c -s`

here directly, like:

RUN echo "deb https://packages.cloud.google.com/apt cloud-sdk-`lsb_release -c -s` main"

great, I will make this update.

bootstraponline · 2016-11-21T19:11:34Z

I think I've addressed all the feedback. Let me know if I missed anything.

viktorbenei · 2016-11-21T19:52:37Z

system_report.sh

@@ -23,6 +23,7 @@ ver_line="$(gradle --version | grep 'Gradle ')" ;     echo "* Gradle: $ver_line"
 ver_line="$(mvn --version | grep 'Apache Maven')" ;   echo "* Maven: $ver_line"
 ver_line="$(fastlane --version | grep 'fastlane ')" ;   echo "* Fastlane: $ver_line"
 ver_line="$( javac -version 2>&1 )" ;                 echo "* Java: $ver_line"
+gcloud version # Lists all gcloud components


as this isn't a one liner, please move it into a separate section like https://github.com/bitrise-docker/android/pull/36/files#diff-1311e47d25cde7d83a0381e5c2b5d06eR32

viktorbenei · 2016-11-21T19:54:14Z

Looks good - hopefully the tests will pass (a full round of test takes ~35 mins right now, as it does a full, clean docker build)

bootstraponline · 2016-11-21T20:43:43Z

Tests passed.

viktorbenei · 2016-11-22T09:33:44Z

LGTM, thank you @bootstraponline !

Release Schedule: as usual, the Android stack will be updated/released during the weekend, your changes will be available at the end of this week.

Thanks for the contribution, and Happy Building! :) 🎉

viktorbenei · 2016-11-22T09:50:19Z

@bootstraponline I made a couple of minor (style) changes, for easier future maintenance - if you have the time feel free to 👍 / 👎 the PR: #37

bootstraponline · 2016-11-22T14:03:19Z

@bootstraponline I made a couple of minor (style) changes, for easier future maintenance - if you have the time feel free to 👍 / 👎 the PR: #37

+1ed.

Thanks for the awesome contribution process. It's great to have rapid feedback & a fast deploy cycle.

viktorbenei · 2016-11-22T14:17:32Z

Thanks for the PR! ;)

Btw technically your change is now available in the -alpha docker image on Docker Hub, but you should not use that on bitrise.io, as those images are not pre cached, and are used only for daily tests.

bootstraponline · 2016-11-22T17:48:41Z

@viktorbenei hey does gcloud info --run-diagnostics work for you? I get this in the container:

root@c9476a01207e:/bitrise/tmp/android#  gcloud info --run-diagnostics
Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
ERROR: Reachability Check failed.
    Cannot reach https://cloudresourcemanager.googleapis.com/v1beta1/projects (ServerNotFoundError)
    Cannot reach https://www.googleapis.com/auth/cloud-platform (ServerNotFoundError)
    Cannot reach https://dl.google.com/dl/cloudsdk/channels/rapid/components-2.json (ServerNotFoundError)
Network connection problems may be due to proxy or firewall settings.

viktorbenei · 2016-11-22T17:59:54Z

That seems like a network config issue on your Mac @bootstraponline - I don't have a VM/machine with the pre release image yet.

bootstraponline · 2016-11-22T18:01:51Z

It works fine outside of the VM so yeah probably Docker is being strange. Hopefully it's not an issue on the production deploy.

viktorbenei · 2016-11-22T18:07:02Z

It shouldn't be, but we'll see - feel free to add additional tests to the system_report, those are run automatically on every change, and daily on the latest image version (alpha)

viktorbenei · 2016-11-22T18:12:29Z

(Additional full test builds are performed on the alpha images before release, first ones on Friday before the "feature lock down")

bootstraponline · 2016-11-22T18:20:11Z

I submitted #38 which runs the network check in system_report.

viktorbenei · 2016-11-23T11:48:14Z

Merged ;)

bootstraponline · 2016-11-23T18:53:37Z

My Android Firebase tests are passing on bitrise. For now I added the install steps from the Dockerfile directly to a workflow. It takes about ~33 seconds to install the google-cloud-sdk package.

It might be nice to have documentation then Google can link people to Bitrise like they do for circle ci.

viktorbenei · 2016-11-23T19:42:39Z

Awesome idea, and thanks for the links @bootstraponline !

I think this could be encapsulated into a Step too, to make it easy to setup & run tests. Feel free to submit a step if you have the time ;)

bootstraponline · 2016-11-23T20:02:01Z

I encapsulated the logic into an internal Fastlane Ruby step. :)

I tested a bash version as well before porting to Ruby. Might be helpful for someone who wants to write a Bitrise Step.

set -euxo pipefail

mkdir -p ~/.gradle

# remove old daemons
rm -rf ~/.gradle/daemon

echo "org.gradle.jvmargs=-Xmx2560M" > ~/.gradle/gradle.properties

./gradlew :app:assembleDebug :app:assembleDebugAndroidTest > /dev/null

# todo: remove this after Dockerfile deploy next week
echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
export DEBIAN_FRONTEND="noninteractive"

echo "deb https://packages.cloud.google.com/apt cloud-sdk-`lsb_release -c -s` main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

sudo apt-get update -qq
sudo apt-get install -y -qq google-cloud-sdk > /dev/null

# now auth to gcloud
echo $GCLOUD_KEY | base64 --decode > "$HOME/gcloudkey.json"
gcloud config set project "$GCLOUD_PROJECT"
gcloud auth activate-service-account --key-file "$HOME/gcloudkey.json" "$GCLOUD_USER"

# run the test!
export APP_APK="/bitrise/src/my/app/build/outputs/apk/app-debug.apk"
export TEST_APK="/bitrise/src/my/app/build/outputs/apk/app-debug-androidTest.apk"

# ensure we don't have old logs
rm -rf gcloud.log

unbuffer \
gcloud beta test android run \
--type instrumentation \
--app "$APP_APK" \
--test "$TEST_APK" \
--results-bucket  my-named-bucket \
--device-ids Nexus9 \
--os-version-ids 25 \
--locales en \
--orientations portrait \
--timeout 25m \
2>&1 | tee gcloud.log

# todo: parse gcloud.log for results bucket link
# and download artifacts with gsutil

viktorbenei · 2016-11-24T08:47:20Z

Thanks @bootstraponline - I created a Bitrise Contrib ticket for this, and linked your comment/code ;) - bitrise-io/bitrise-contrib#29

Add Google Cloud SDK

c8af605

viktorbenei suggested changes Nov 21, 2016

View reviewed changes

Don't change HOME

baf9fe6

bootstraponline added 5 commits November 21, 2016 13:49

Install via apt

9232035

Install recommended packages also

5b4d7bb

Add doc link

d8a1f7b

Set CLOUD_SDK_REPO

34a8da8

Disable usage reporting

20105ec

viktorbenei reviewed Nov 21, 2016

View reviewed changes

Update system report

173ecb7

viktorbenei reviewed Nov 21, 2016

View reviewed changes

Avoid hard coding the Ubuntu distro

4a935a7

Add comment about config.json

b6b8c5e

viktorbenei reviewed Nov 21, 2016

View reviewed changes

Move gcloud to new section

5ddf405

viktorbenei approved these changes Nov 22, 2016

View reviewed changes

viktorbenei merged commit 0bed835 into bitrise-io:master Nov 22, 2016

bootstraponline deleted the firebase_test_lab branch November 22, 2016 14:00

viktorbenei mentioned this pull request Nov 24, 2016

Google Firebase Android testing service step bitrise-io/bitrise-contrib#29

Closed


		ENV CLOUD_SDK_REPO="cloud-sdk-xenial"

		RUN echo "deb https://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" \| sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list \

Add Google Cloud SDK #36

Add Google Cloud SDK #36

Conversation

bootstraponline commented Nov 21, 2016 • edited Loading

viktorbenei commented Nov 21, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bootstraponline Nov 21, 2016 • edited Loading

Choose a reason for hiding this comment

viktorbenei commented Nov 21, 2016

bootstraponline commented Nov 21, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bootstraponline commented Nov 21, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bootstraponline commented Nov 21, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viktorbenei commented Nov 21, 2016

bootstraponline commented Nov 21, 2016

viktorbenei commented Nov 22, 2016

viktorbenei commented Nov 22, 2016

bootstraponline commented Nov 22, 2016

viktorbenei commented Nov 22, 2016

bootstraponline commented Nov 22, 2016

viktorbenei commented Nov 22, 2016

bootstraponline commented Nov 22, 2016

viktorbenei commented Nov 22, 2016

viktorbenei commented Nov 22, 2016

bootstraponline commented Nov 22, 2016

viktorbenei commented Nov 23, 2016

bootstraponline commented Nov 23, 2016

viktorbenei commented Nov 23, 2016

bootstraponline commented Nov 23, 2016 • edited Loading

viktorbenei commented Nov 24, 2016

bootstraponline commented Nov 21, 2016 •

edited

Loading

bootstraponline Nov 21, 2016 •

edited

Loading

bootstraponline commented Nov 23, 2016 •

edited

Loading