add test suite based on conu library #167

TomasTomecek · 2018-01-16T10:17:38Z

Let's discuss the invocation please. I'm not entirely sure how to integrate it into your makefile hierarchy.

The library: https://github.com/fedora-modularity/conu

Signed-off-by: Tomas Tomecek ttomecek@redhat.com

praiskup · 2018-01-16T10:57:43Z

Explicit test make target seems to be fine, if this is going to be shared among containers then it should be in common.mk.

Explicit test make target seems to be fine to me, if this is going to be shared among containers then it should belong to common.mk.. so @omron93 can run those tests in our jenkins(es).

pkubatrh · 2018-01-16T11:23:24Z

It would be nice if the test cases lived inside the global test directory and were able to test all versions of ruby.

pvalena · 2018-01-16T11:46:08Z

Makefile

@@ -1,5 +1,7 @@
 # Variables are documented in common/build.sh.
 BASE_IMAGE_NAME = ruby
+TEST_IMAGE_NAME := $(BASE_IMAGE_NAME)-tests
+TEST_TARGET := 2.4/test/test_s2i.py


Yes, it would be nice to test all in an automated way. And also:

# Variables are documented in common/build.sh.

omron93 · 2018-01-22T09:13:29Z

Explicit test make target seems to be fine to me, if this is going to be shared among containers then it should belong to common.mk.. so @omron93 can run those tests in our jenkins(es).

Run these tests explicitly? (separate make target)
Or as part of make test or make check ?

IMHO If we want to move logic for conu tests in read-only common dir, then test targets have to be stored in a file in image repo. What's the best name? Or is there any other way how to list all conu tests to run?

TomasTomecek · 2018-01-22T09:52:32Z

I won't have time to work on this PR in the current week. At the same time, I'm also waiting for you to agree on structure and will follow with the updates afterwards.

hhorak · 2018-01-27T12:55:37Z

2.4/test/test_s2i.py

+        finally:
+            c.stop()
+            c.wait()
+            c.delete()


This workflow of wait, assert, stop, wait, delete seems to be to be used pretty often. We might have a method image.check_output(["ruby", "--version"], "ruby 2.4."). And the whole test case could be shortened to:

backend.ImageClass(image_name).check_output(command=["ruby", "--version"], matches="*ruby 2.4.*")

...or something along those lines..

Related: sclorg/s2i-ruby-container#167 Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

also in multiple environments Related: sclorg/s2i-ruby-container#167 Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

centos-ci · 2018-04-16T13:27:56Z

Can one of the admins verify this patch?

pvalena · 2018-04-16T13:44:21Z

2.3/test/test_s2i.py

@@ -0,0 +1 @@
+../../2.4/test/test_s2i.py


This is, I'd say, why we have container-common-scripts repo. I'd rather not symlink between versions.

Or just put it into $ROOT/test and symlink from the versioned directories.

pvalena · 2018-04-16T13:45:58Z

Makefile

@@ -1,5 +1,6 @@
 # Variables are documented in common/build.sh.
 BASE_IMAGE_NAME = ruby
+TEST_SUITES=test/run-conu


This would effectively disable test/run test suite, right?

I'm actually really fuzzy about your scripts and most of the time I'm like

So yeah, it seems it disables it. Originally I wanted to move it to $ROOT/test but I realized I don't know how to do it.

I think it is ok the way it is. Just change the assignment to append the new test suite instead and move it so that it is appended after common.mk is included

pkubatrh · 2018-04-17T06:53:22Z

2.4/test/run-conu

@@ -0,0 +1,2 @@
+#!/bin/bash
+exec pytest-3 -vv ./test_s2i.py


This needs to be: exec pytest-3 -vv ./test/test_s2i.py
The test is being run from $ROOT/$VERSION

pkubatrh · 2018-04-17T08:49:00Z

2.4/test/test_s2i.py

+            assert c.is_port_open(8080)
+            response = c.http_request("/", port="8080")
+            assert response.ok
+            output = c.execute(["ruby", "--version"])


This needs to be ["bash", "-c", "ruby --version"]

pkubatrh · 2018-04-17T09:01:33Z

2.4/test/test_s2i.py

+            logs = c.logs().decode("utf-8").strip()
+            usage = i.usage()
+            # FIXME: workaround: `docker logs` can't handle logs like these: '\n\n\n'
+            assert logs.replace("\n", "") == usage.replace("\n", "")


A failure over here since the usage call does not ignore stderr

Actually the usage call only returns stderr ...

pvalena · 2018-09-14T11:34:19Z

This branch has conflicts that must be resolved

TomasTomecek · 2018-09-17T08:16:29Z

I'll be honest: I don't have time to finish this anymore.

Is this merge-able if I just rebase or is there anything else what needs to be done?

pvalena · 2018-09-19T13:44:06Z

There are comments in the code regarding fixes needed- AFAICT just implementation details.
I think we can fix those ourselves.
Thanks!

TomasTomecek · 2018-11-16T16:32:50Z

Finally found some spare time to finish this. Please see my changes.

Obviously this requires sclorg/container-common-scripts#67

TomasTomecek · 2018-11-20T15:08:46Z

sclorg/container-common-scripts#67 is merged, I updated this PR and is ready for review

pkubatrh · 2018-11-26T12:35:30Z

Do we have a way to test this in CI now? afaics the conu tests are run only via make test-with-conu

TomasTomecek · 2018-11-26T14:40:45Z

You can probably create a new job and run that command and see if it works.

Please let me know what I should do in order to get this merged.

omron93 · 2018-11-28T13:31:43Z

I've updated CI config for ruby repo [1]. make test-with-conu is run at same time as make test, so triggered by [test]. Different ideas are welcomed...

Because elel7 builds are quite outdated, conu is installed from PyPi (docker-squash is already installed from PyPi, so nothing new).
-> pytest is also installed from PyPi into virtualenv

[test-openshift]

[1] sclorg/rhscl-container-ci#19

omron93 · 2018-11-28T13:34:17Z

test/run-conu

@@ -0,0 +1,2 @@
+#!/bin/bash
+exec pytest-3 -vv -k test_s2i


@TomasTomecek Please use pytest. If conu is installed from PyPi, pytest is also from PyPi and no -3 suffix is there. (to note: although without -3, python3 is used)

omron93 · 2018-11-28T13:38:43Z

EDIT: Ignore the bellow. It was caused by docker vs docker-py module conflict when using docker-squash==1.0.5.

@TomasTomecek Also when I've tried the same (conu from PyPi with docker-squash==1.0.5) I this error. Any idea?

=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.6.6, pytest-4.0.1, py-1.7.0, pluggy-0.8.0 -- /home/mskalick/pypi/bin/python3
cachedir: .pytest_cache
rootdir: /home/mskalick/Packages/docker/sclorg/s2i-ruby-container/2.3, inifile:
collected 0 items / 1 errors                                                                                                                                                                                      

===================================================================================================== ERRORS ======================================================================================================
________________________________________________________________________________________ ERROR collecting test/test_s2i.py ________________________________________________________________________________________
test/test_s2i.py:21: in <module>
    backend = DockerBackend(logging_level=logging.DEBUG)
/home/mskalick/pypi/lib/python3.6/site-packages/conu/backend/docker/backend.py:75: in __init__
    self.d = get_client()
/home/mskalick/pypi/lib/python3.6/site-packages/conu/backend/docker/client.py:37: in get_client
    client = docker.APIClient(version="auto")  # >= 2
/home/mskalick/pypi/lib/python3.6/site-packages/docker/api/client.py:113: in __init__
    config_dict=self._general_configs
E   TypeError: load_config() got an unexpected keyword argument 'config_dict'
------------------------------------------------------------------------------------------------- Captured stderr -------------------------------------------------------------------------------------------------
15:01:05.000 backend.py        INFO   conu has initiated, welcome to the party!
15:01:05.000 backend.py        DEBUG  conu version: 0.6.2
15:01:05.109 __init__.py       INFO   docker environment info: 'Client:\n Version:         1.13.1\n API version:     1.26\n Package version: docker-1.13.1-60.git9cb56fd.fc27.x86_64\n Go version:      go1.9.7\n Git commit:      9c9378f-unsupported\n Built:           Sun Jul  8 08:52:30 2018\n OS/Arch:         linux/amd64\n\nServer:\n Version:         1.13.1\n API version:     1.26 (minimum version 1.12)\n Package version: docker-1.13.1-60.git9cb56fd.fc27.x86_64\n Go version:      go1.9.7\n Git commit:      9c9378f-unsupported\n Built:           Sun Jul  8 08:52:30 2018\n OS/Arch:         linux/amd64\n Experimental:    false\n'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================================= 1 error in 0.88 seconds =============================================================================================
make: *** [common/common.mk:93: test-with-conu] Chyba 2

omron93 · 2018-11-30T14:06:54Z

@TomasTomecek CI is ready. So only pytest-3 is blocking it to run...

When trying this locally, I'm getting this test failure:

test/test_s2i.py::TestSuite::test_usage FAILED                                                                                                                                                              [100%]

==================================================================================================== FAILURES =====================================================================================================
______________________________________________________________________________________________ TestSuite.test_usage _______________________________________________________________________________________________

self = <test_s2i.TestSuite object at 0x7f0ef74a4e10>

    def test_usage(self):
        i = S2IDockerImage(image_name)
        c = i.run_via_binary()
    
        def logs_received():
            return len(list(c.logs())) > 0
    
        try:
            c.wait()
            # even after waiting there is still a race in journal logging driver
            Probe(timeout=10, pause=0.05, count=20, fnc=logs_received).run()
            logs = [x.decode("utf-8") for x in c.logs()]
            logs = "\n".join(logs).strip()
            usage = i.usage()
            # FIXME: workaround: `docker logs` can't handle logs like these: '\n\n\n'
>           assert logs.replace("\n", "") == usage.replace("\n", "")
E           assert 'This is a S2...by-sample-app' == 'I1130 15:03:4...ec/s2i/usage"'
E             - This is a S2I ruby-2.5 rhel base image:To use it, install S2I: https://github.com/openshift/source-to-imageSample invocation:s2i build https://github.com/sclorg/s2i-ruby-container.git --context-dir=2.5/test/puma-test-app/ rhscl/ruby-25-rhel7 ruby-sample-appYou can then run the resulting image via:docker run -p 8080:8080 ruby-sample-app
E             + I1130 15:03:40.440308 29185 install.go:251] Using "usage" installed from "image:///usr/libexec/s2i/usage"

test/test_s2i.py:81: AssertionError
---------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------
15:03:39.995 image.py          INFO   run container via binary in background
15:03:39.996 image.py          DEBUG  docker command: ['docker', 'run', '-d', '--cidfile=/tmp/conu-qm0_epmt/conu-fxskgnfdytoqxoxvppuunildfberbdxs', '-l', 'conu.test_artifact', 'sha256:8f849caf8b79374d20b5d0968008119ec769891fcd9a7674d97c2d5b4537a927']
15:03:39.996 __init__.py       DEBUG  command: "docker run -d --cidfile=/tmp/conu-qm0_epmt/conu-fxskgnfdytoqxoxvppuunildfberbdxs -l conu.test_artifact sha256:8f849caf8b79374d20b5d0968008119ec769891fcd9a7674d97c2d5b4537a927"
15:03:40.239 __init__.py       DEBUG  c22ba48032bb2d366c21dc625a0f2705cb8ebba20cc7da05a226fca170b9c52e

15:03:40.240 probes.py         DEBUG  starting probe
15:03:40.242 probes.py         DEBUG  first process started: pid=29157
15:03:40.243 probes.py         DEBUG  pausing for 0.1 before next try
15:03:40.254 probes.py         DEBUG  Running "get_cont_id" with parameters: "{}":	0/2
15:03:40.255 probes.py         DEBUG  callback result = True
15:03:40.348 probes.py         DEBUG  waiting for process to end...
15:03:40.349 probes.py         DEBUG  result = True
15:03:40.362 probes.py         DEBUG  starting probe
15:03:40.364 probes.py         DEBUG  first process started: pid=29183
15:03:40.365 probes.py         DEBUG  pausing for 0.05 before next try
15:03:40.367 probes.py         DEBUG  Running "logs_received" with parameters: "{}":	0/10
15:03:40.379 probes.py         DEBUG  callback result = True
15:03:40.415 probes.py         DEBUG  waiting for process to end...
15:03:40.416 probes.py         DEBUG  result = True

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

TomasTomecek · 2018-12-04T18:05:39Z

@omron93 I got an error like this when using older s2i -- newer s2i shouldn't print this debug output: I1130 15:03:4

I just updated the command to be pytest and force-pushed.

(got sick, hence the late response)

omron93 · 2018-12-05T09:42:28Z

[test][test-openshift]

running conu tests (sclorg/s2i-ruby-container/pull/167) IMHO, the best way to solve is to do epel build (epel is enabled already)

omron93 · 2018-12-06T11:39:04Z

Newer s2i worked. So tests are passing now.

This PR got a deep review, so merging. @TomasTomecek Thanks for adding this feature.

TomasTomecek · 2018-12-06T16:17:33Z

It was almost a year! I wanted to have this merged so it wouldn't have to be my new year resolution. Thank you to everyone who got involved and helped carrying me.

* Set defaults for lang/locale and I/O encoding. * Fix use of wrong locale. Was AU instead of US.

pvalena reviewed Jan 16, 2018

View reviewed changes

hhorak mentioned this pull request Jan 27, 2018

[WIP] First try of conu test implementation sclorg/s2i-php-container#198

Closed

hhorak reviewed Jan 27, 2018

View reviewed changes

TomasTomecek added a commit to TomasTomecek/container-common-scripts that referenced this pull request Feb 20, 2018

add a way to run conu tests in a container

a4948c1

Related: sclorg/s2i-ruby-container#167 Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

TomasTomecek mentioned this pull request Feb 20, 2018

add a way to run multiple test suites sclorg/container-common-scripts#67

Closed

4 tasks

TomasTomecek added a commit to TomasTomecek/container-common-scripts that referenced this pull request Mar 8, 2018

add a way to run multiple test suites

a7e8a3a

also in multiple environments Related: sclorg/s2i-ruby-container#167 Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

TomasTomecek added a commit to TomasTomecek/container-common-scripts that referenced this pull request Mar 8, 2018

add a way to run multiple test suites

be5c88a

also in multiple environments Related: sclorg/s2i-ruby-container#167 Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

TomasTomecek force-pushed the add-conu-tests branch from 00b71a4 to 35f64c7 Compare March 29, 2018 14:08

TomasTomecek force-pushed the add-conu-tests branch from 35f64c7 to 43e0d9d Compare April 16, 2018 13:27

pvalena reviewed Apr 16, 2018

View reviewed changes

pkubatrh reviewed Apr 17, 2018

View reviewed changes

sclorg deleted a comment from rhscl-bot Sep 14, 2018

sclorg deleted a comment from centos-ci Sep 19, 2018

sclorg deleted a comment from rhscl-bot Sep 19, 2018

TomasTomecek force-pushed the add-conu-tests branch from 43e0d9d to 53bde92 Compare November 16, 2018 16:31

TomasTomecek force-pushed the add-conu-tests branch from 53bde92 to 085e7fd Compare November 20, 2018 15:07

omron93 reviewed Nov 28, 2018

View reviewed changes

add test suite based on conu library

d09d821

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

TomasTomecek force-pushed the add-conu-tests branch from 085e7fd to d09d821 Compare December 4, 2018 18:05

omron93 merged commit 06867bb into sclorg:master Dec 6, 2018

TomasTomecek deleted the add-conu-tests branch December 6, 2018 16:16

hhorak pushed a commit to hhorak/s2i-ruby-container that referenced this pull request Aug 13, 2020

Set defaults for lang/locale and I/O encoding. (sclorg#167)

03dc173

* Set defaults for lang/locale and I/O encoding. * Fix use of wrong locale. Was AU instead of US.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add test suite based on conu library #167

add test suite based on conu library #167

TomasTomecek commented Jan 16, 2018

praiskup commented Jan 16, 2018

pkubatrh commented Jan 16, 2018 •

edited

Loading

pvalena Jan 16, 2018

omron93 commented Jan 22, 2018

TomasTomecek commented Jan 22, 2018

hhorak Jan 27, 2018

centos-ci commented Apr 16, 2018

pvalena Apr 16, 2018

pkubatrh Apr 17, 2018

pvalena Apr 16, 2018

TomasTomecek Apr 16, 2018

pkubatrh Apr 17, 2018

pkubatrh Apr 17, 2018

pkubatrh Apr 17, 2018

pkubatrh Apr 17, 2018

pkubatrh Apr 17, 2018

pvalena commented Sep 14, 2018

TomasTomecek commented Sep 17, 2018

pvalena commented Sep 19, 2018 •

edited

Loading

TomasTomecek commented Nov 16, 2018

TomasTomecek commented Nov 20, 2018

pkubatrh commented Nov 26, 2018

TomasTomecek commented Nov 26, 2018

omron93 commented Nov 28, 2018

omron93 Nov 28, 2018

omron93 commented Nov 28, 2018 •

edited

Loading

omron93 commented Nov 30, 2018

TomasTomecek commented Dec 4, 2018

omron93 commented Dec 5, 2018

omron93 commented Dec 6, 2018

TomasTomecek commented Dec 6, 2018

		@@ -0,0 +1 @@
		../../2.4/test/test_s2i.py

add test suite based on conu library #167

add test suite based on conu library #167

Conversation

TomasTomecek commented Jan 16, 2018

praiskup commented Jan 16, 2018

pkubatrh commented Jan 16, 2018 • edited Loading

Choose a reason for hiding this comment

omron93 commented Jan 22, 2018

TomasTomecek commented Jan 22, 2018

Choose a reason for hiding this comment

centos-ci commented Apr 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pvalena commented Sep 14, 2018

TomasTomecek commented Sep 17, 2018

pvalena commented Sep 19, 2018 • edited Loading

TomasTomecek commented Nov 16, 2018

TomasTomecek commented Nov 20, 2018

pkubatrh commented Nov 26, 2018

TomasTomecek commented Nov 26, 2018

omron93 commented Nov 28, 2018

Choose a reason for hiding this comment

omron93 commented Nov 28, 2018 • edited Loading

omron93 commented Nov 30, 2018

TomasTomecek commented Dec 4, 2018

omron93 commented Dec 5, 2018

omron93 commented Dec 6, 2018

TomasTomecek commented Dec 6, 2018

pkubatrh commented Jan 16, 2018 •

edited

Loading

pvalena commented Sep 19, 2018 •

edited

Loading

omron93 commented Nov 28, 2018 •

edited

Loading