use newer certmanager #1298

ryandawsonuk · 2019-12-20T12:26:14Z

seldondev · 2019-12-20T12:27:26Z

Fri Dec 20 12:27:25 UTC 2019
The logs for [pr-build] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=1

ryandawsonuk · 2019-12-20T12:27:28Z

/ok-to-test
/test integration

seldondev · 2019-12-20T12:27:38Z

Fri Dec 20 12:27:37 UTC 2019
The logs for [lint] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=2

seldondev · 2019-12-20T12:28:52Z

Fri Dec 20 12:28:51 UTC 2019
The logs for [integration] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=4

seldondev · 2019-12-20T12:29:00Z

Fri Dec 20 12:28:58 UTC 2019
The logs for [pr-build] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=3

seldondev · 2019-12-20T12:29:14Z

Fri Dec 20 12:29:13 UTC 2019
The logs for [lint] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=5

ryandawsonuk · 2019-12-20T12:54:58Z

/test integration

seldondev · 2019-12-20T12:56:11Z

Fri Dec 20 12:56:10 UTC 2019
The logs for [integration] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=6

ryandawsonuk · 2019-12-20T13:25:25Z

Hmm, the e2e errors I'm seeing from server are no matches for kind "SeldonDeployment" in version "machinelearning.seldon.io/v1alpha2". But the tests all pass locally against kind.

ukclivecox · 2019-12-20T13:29:14Z

/approve

seldondev · 2019-12-20T13:29:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cliveseldon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cliveseldon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ryandawsonuk · 2019-12-20T14:54:09Z

Problem in the cluster is coming from the istio rollout

Waiting for deployment "istio-ingressgateway" rollout to finish: 0 of 1 updated replicas are available...
error: deployment "istio-ingressgateway" exceeded its progress deadline
make: *** [Makefile:39: install_istio] Error 1

ryandawsonuk · 2019-12-20T14:56:38Z

/test integration

seldondev · 2019-12-20T15:01:53Z

Fri Dec 20 15:01:51 UTC 2019
The logs for [integration] [7] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/7.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=7

seldondev · 2019-12-20T15:35:09Z

Fri Dec 20 15:35:08 UTC 2019
The logs for [pr-build] [8] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/8.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=8

seldondev · 2019-12-20T15:35:12Z

Fri Dec 20 15:35:11 UTC 2019
The logs for [lint] [9] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/9.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=9

seldondev · 2019-12-20T15:36:48Z

Fri Dec 20 15:36:47 UTC 2019
The logs for [pr-build] [10] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/10.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=10

seldondev · 2019-12-20T15:36:50Z

Fri Dec 20 15:36:49 UTC 2019
The logs for [lint] [11] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/11.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=11

ryandawsonuk · 2019-12-20T15:38:35Z

/test integration

seldondev · 2019-12-20T15:39:56Z

Fri Dec 20 15:39:55 UTC 2019
The logs for [integration] [12] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/12.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=12

ryandawsonuk · 2019-12-20T15:55:13Z

The problem with the rollout status on istio-ingressgateway seems to be that you can eventually lose the watch.

I did an exec into the container and pointed my shell at the kind context inside. This revealed several stuck istio pods:

I then looked at the logs for the istio-galley pod and found the error:

2019-12-20T15:51:10.410272Z     fatal   Error creating server: no space left on device

So I guess the problem is insufficient ephemeral storage

ryandawsonuk · 2019-12-20T16:13:24Z

Have removed old jobs that may be using disk space. Trying again.

ryandawsonuk · 2019-12-20T16:13:31Z

/test integration

seldondev · 2019-12-20T16:14:47Z

Fri Dec 20 16:14:46 UTC 2019
The logs for [integration] [13] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/13.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=13

ryandawsonuk · 2019-12-20T16:57:59Z

After removing all the old completed pods, the only failure was the usual intermittent 503 problem.

seldondev · 2019-12-20T16:59:45Z

Fri Dec 20 16:59:44 UTC 2019
The logs for [lint] [15] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/15.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=15

seldondev · 2019-12-20T16:59:50Z

Fri Dec 20 16:59:49 UTC 2019
The logs for [pr-build] [14] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/14.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=14

ryandawsonuk · 2019-12-20T17:04:24Z

/test integration

seldondev · 2019-12-20T17:05:47Z

Fri Dec 20 17:05:46 UTC 2019
The logs for [integration] [16] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/16.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=16

ryandawsonuk · 2019-12-20T17:30:45Z

Now seeing failed to watch file "/var/lib/docker/containers/087ca3727bd3bb066021cf4f2462441627d90ba6c8d0abbc4bf6a0910b96318f/087ca3727bd3bb066021cf4f2462441627d90ba6c8d0abbc4bf6a0910b96318f-json.log": no space left on device(base) inside the build container logs themselves. Could be related to jenkins-x/jx#5622

However it did eventually finish and again only one failure... an intermittent ambassador 503 in one of the rolling update tests (this time test_rolling_update2)

ryandawsonuk · 2019-12-20T17:49:56Z

/test integration

seldondev · 2019-12-20T17:51:12Z

Fri Dec 20 17:51:11 UTC 2019
The logs for [integration] [17] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1298/17.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1298 --build=17

use newer certmanager

e74dcce

seldondev requested review from axsaucedo and ukclivecox December 20, 2019 12:26

seldondev added the size/M label Dec 20, 2019

seldondev added the ok-to-test label Dec 20, 2019

seldondev added the approved label Dec 20, 2019

ryandawsonuk added 2 commits December 20, 2019 15:33

enable cert-manager with kubeflow

d0cf3bf

see if more resource allows e2e to run istio

4a36838

put resources back as they were

ab2dfd0

seldondev merged commit 2e06cd6 into SeldonIO:master Dec 20, 2019

ryandawsonuk mentioned this pull request Jan 8, 2020

Internal build issue: PR builds intermittently fail with no space left on device #1322

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use newer certmanager #1298

use newer certmanager #1298

ryandawsonuk commented Dec 20, 2019 •

edited

Loading

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019 •

edited

Loading

ukclivecox commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019 •

edited

Loading

ryandawsonuk commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019 •

edited

Loading

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

use newer certmanager #1298

use newer certmanager #1298

Conversation

ryandawsonuk commented Dec 20, 2019 • edited Loading

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019 • edited Loading

ukclivecox commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019 • edited Loading

ryandawsonuk commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019 • edited Loading

ryandawsonuk commented Dec 20, 2019

seldondev commented Dec 20, 2019

ryandawsonuk commented Dec 20, 2019 •

edited

Loading

ryandawsonuk commented Dec 20, 2019 •

edited

Loading

ryandawsonuk commented Dec 20, 2019 •

edited

Loading

ryandawsonuk commented Dec 20, 2019 •

edited

Loading