Large tests and runnable example

[marcin-krolik]

• runnable example added
• large tests implemented in python

[marcin-krolik]

PR #19 has to be merged before this one in order to enable downloading plugin binary from proper path at S3

[kindermoumoute]

Add f and L flags for handling errors in the curl. ... curl -sfLSO ...

[marcin-krolik]

Fixed

[kindermoumoute]

The URL will change to http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-meminfo/latest_build/linux/x86_64/snap-plugin-collector-meminfo. #22

[marcin-krolik]

Goo catch, missed it somehow

[kindermoumoute]

When I run the test I got an error. Here is the full log:

❯ make test-large
bash -c "./scripts/test.sh large"
/Users/ocano/work/src/github.com/intelsdi-x/snap-plugin-collector-meminfo/scripts/docker/large
main uses an image, skipping
/Users/ocano/work/src/github.com/intelsdi-x/snap-plugin-collector-meminfo/scripts/docker/large
Creating meminfo_large_test
Attaching to meminfo_large_test
meminfo_large_test | 2016-09-22 06:50:47 [ INFO ] Downloading http://snap.ci.snap-telemetry.io/snap/master/latest/snapd to /usr/local/bin
meminfo_large_test | 2016-09-22 06:50:53 [ INFO ] Downloading http://snap.ci.snap-telemetry.io/snap/master/latest/snapctl to /usr/local/bin
meminfo_large_test | 2016-09-22 06:50:56 [ INFO ] Downloading http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-meminfo/latest_build/linux/x86_64/snap-plugin-collector-meminfo to /etc/snap/plugins
meminfo_large_test | 2016-09-22 06:50:58 [ INFO ] Downloading http://snap.ci.snap-telemetry.io/snap/master/latest/snap-plugin-publisher-mock-file to /etc/snap/plugins
meminfo_large_test | 2016-09-22 06:51:01 [ INFO ] starting snapd
meminfo_large_test | 2016-09-22 06:51:02 [ INFO ] snapctl plugin load /etc/snap/plugins/snap-plugin-collector-meminfo
meminfo_large_test | 2016-09-22 06:51:03 [ INFO ] snapctl plugin load /etc/snap/plugins/snap-plugin-publisher-mock-file
meminfo_large_test | 2016-09-22 06:51:05 [ INFO ] snapctl task create -t /snap-plugin-collector-meminfo/examples/tasks/task-mem.json
meminfo_large_test | 2016-09-22 06:51:16 [ INFO ] snapctl task stop 2f609153-4116-41f5-9fb9-84c234921076
meminfo_large_test | 2016-09-22 06:51:16 [ INFO ] snapctl plugin unload collector:meminfo:3
meminfo_large_test | F2016-09-22 06:51:16 [ INFO ] stopping snapd
meminfo_large_test |
meminfo_large_test | ======================================================================
meminfo_large_test | FAIL: test_meminfo_collector_plugin (__main__.MemInfoCollectorLargeTest)
meminfo_large_test | ----------------------------------------------------------------------
meminfo_large_test | Traceback (most recent call last):
meminfo_large_test |   File "/snap-plugin-collector-meminfo/scripts/test/large.py", line 104, in test_meminfo_collector_plugin
meminfo_large_test |     self.assertEqual(len(metrics), 0, "Metrics available {} expected {}".format(len(metrics), 0))
meminfo_large_test | AssertionError: Metrics available 1 expected 0
meminfo_large_test |
meminfo_large_test | ----------------------------------------------------------------------
meminfo_large_test | Ran 1 test in 28.241s
meminfo_large_test |
meminfo_large_test | FAILED (failures=1)
meminfo_large_test exited with code 1
exit code from large_jenkins 1
/Users/ocano/work/src/github.com/intelsdi-x/snap-plugin-collector-meminfo/scripts/docker/large
/Users/ocano/work/src/github.com/intelsdi-x/snap-plugin-collector-meminfo/scripts/docker/large
Going to remove meminfo_large_test
Removing meminfo_large_test ... done
make: *** [test-large] Error 1

Example is working.

[marcin-krolik]

@kindermoumoute this was due to changes introduces in snap framework, specific to snapctl metric list output on empty metric catalog. Fix was applied to snap-pytest.

Nevertheless test will fail with latest snap build. Reason behind it is this bug

[kindermoumoute]

The goal of doing those large tests is to make it run on K8S. For that you need to add a deployment file, and large_k8s.sh that will run by running TEST_K8S=1 make test-large.

Example:
https://github.com/intelsdi-x/snap-plugin-publisher-influxdb/blob/master/scripts/config/influxdb-deployment.yml
https://github.com/intelsdi-x/snap-plugin-publisher-influxdb/blob/master/scripts/large_k8s.sh

[marcin-krolik]

@kindermoumoute I don't think it's needed to add large_k8s.sh file. I discussed it with @jcooklin before and only deployment yaml file is needed along with proper configuration pushed to jenkins-job-configs. large_k8s.sh was only needed to test large tests running on k8s localy

[kindermoumoute]

When I run test for K8S, there is a permission denied error:

❯ TEST_K8S=1 make test-large
bash -c "./scripts/test.sh large"
2016-10-03 17:16:57 UTC [     info] updating repository submodule with pytest
2016-10-03 17:16:57 UTC [     info] execute large test
2016-10-03 10:16:57 [ INFO ] Downloading http://snap.ci.snap-telemetry.io/snap/master/latest/snapd to /usr/local/bin
2016-10-03 10:17:06 [ INFO ] Downloading http://snap.ci.snap-telemetry.io/snap/master/latest/snapctl to /usr/local/bin
2016-10-03 10:17:07 [ INFO ] Downloading http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-meminfo/latest_build/linux/x86_64/snap-plugin-collector-meminfo to /etc/snap/plugins
2016-10-03 10:17:07 [ ERROR ] 13
E
======================================================================
ERROR: test_meminfo_collector_plugin (__main__.MemInfoCollectorLargeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ocano/work/src/github.com/intelsdi-x/snap-plugin-collector-meminfo/scripts/test/large.py", line 45, in setUp
    utils.download_binaries(self.binaries)
  File "/Users/ocano/work/src/github.com/intelsdi-x/snap-plugin-collector-meminfo/scripts/test/pytest/utils.py", line 28, in download_binaries
    _download_binary(binary)
  File "/Users/ocano/work/src/github.com/intelsdi-x/snap-plugin-collector-meminfo/scripts/test/pytest/utils.py", line 45, in _download_binary
    _ensure_dir(binary.dir)
  File "/Users/ocano/work/src/github.com/intelsdi-x/snap-plugin-collector-meminfo/scripts/test/pytest/utils.py", line 33, in _ensure_dir
    os.makedirs(dirname)
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/etc/snap'

----------------------------------------------------------------------
Ran 1 test in 9.879s

FAILED (errors=1)

[kindermoumoute]

More a general comment, using a python framework here makes it harder for developer to understand plugin codebase:

• There are already Go and Shell, adding a 3rd language might be a barrier.
• Codebase is bigger and there is duplicate code with example scripts, might be confusing.

Which bring us to ask those question:

• Do we want developer to deal with large test framework?
• Can we make a unified and consistency test framework for large test?

This would bring us to have an agreement on what we want to test for large test. This PR is testing more cases than other existing large tests, which is really cool.

• Do we want plugin developers to define those cases?
• Can we agree on what large tests will test for plugins?

I would be interested to have your thoughts on those questions.

Also, you can take a look on a proposal I made here to have the current test framework synchronized with pluginsync. This is one solution, I believe there are other solutions.

[marcintao]

LGTM

[marcin-krolik]

Do we want developer to deal with large test framework?

If we plan to assure quality and stability of plugins and framework I think we need a framework for such kind of tests. It's going to introduce new code base and need to maintain it. I don't know of any other way to achieve reasonable quality of whole product.

Can we make a unified and consistency test framework for large test?

Sure. What I already prepared is a example of such framework.

Do we want plugin developers to define those cases?

I think it more of a question "When plugin is complete?". Is it enough to provide unit tests and documentation? Or maybe runnable example as well? How about large tests?

[marcin-krolik]

@kindermoumoute As for your comment regarding large_k8s.sh. This script is not for local test. It is pointed in jenkins job configuration as the one to execute. If you would like to test kubernetes deployment file you have to do it manually executing kubectl create -f k8s-deply.yml and so on. What I understood from discussion with @jcooklin there is no need to create script to execute large tests locally on kubernetes in order to implement large tests.

[kindermoumoute]

LGTM