-
Notifications
You must be signed in to change notification settings - Fork 313
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run integration tests with pytest #919
Conversation
With this commit we add a proof-of-concept implementation of how integration tests could look like with py.test instead of shell scripts. For the proof of concept we migrate the integration tests for the list subcommands and source builds. It also spins up an Elasticsearch metrics store and checks that any preconditions (basically Docker being present and up and running) are met. Finally, it also integrates into CI by writing JUnit XML files for test results. Relates elastic#736
Unfortunately we don't see any Rally output in the PR checks. Therefore I have raised #923. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work and foundation!
I love the iterative approach (i.e. gradually replace the functionality in integration-test.sh
) and the fact that this is so readable and easy to read makes me happy.
I left a comment/suggestion to use Template
rather than replacing strings (and a FYI comment).
def copy_config(name): | ||
f = ConfigFile(name) | ||
io.ensure_dir(f.rally_home) | ||
with open(f.target_path, "w", encoding="UTF-8") as target: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it make sense to just read src into contents
with something:
with open(f.source_path, "r", encoding="UTF-8") as src:
contents = src.read()
and then replace using Template
's native capabilities, to avoid pure string based replacements, i.e.
from string import Template
with open(f.target_path, "w", encoding="UTF-8") as target:
target.write(Template(contents).substitute(USER_HOME=f.user_name)
AFAIK ${identifier}
-like template strings are supported out of the box.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Much better. Addressed in eb03450.
return | ||
else: | ||
c.close() | ||
time.sleep(0.5) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not directly related to this PR, but going forwards should we start embracing the https://github.com/jd/tenacity library, which seems to be becoming the standard in retrying in Python? It offers a lot flexibility in what to retry with boolean operations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we have some timeout here just so our it tests dont run forever on a PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am reluctant adding additional dependencies for such small pieces of functionality but I added a two minute timeout in 3537d0b.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is great work, LGTM. nothing but super minor nits.
return | ||
else: | ||
c.close() | ||
time.sleep(0.5) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we have some timeout here just so our it tests dont run forever on a PR?
it/__init__.py
Outdated
if process.run_subprocess_with_logging("docker") != 0: | ||
raise AssertionError("Docker is required to run integration tests.") | ||
if process.run_subprocess_with_logging("docker ps") != 0: | ||
raise AssertionError("Docker daemon must be up and running to run integration tests.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need both of these checks? We can just say "docker must be installed and running" if docker ps
fails
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. I combined the checks in 7786bdc.
|
||
class ConfigFile: | ||
def __init__(self, config_name): | ||
self.user_home = os.path.expanduser("~") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe use pathlib's Path.home()
instead of the ~
since tilde expansion is black magic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd tend to be consistent across our code base and keep this idiom.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Thanks for your reviews! |
With this commit we add a proof-of-concept implementation of how
integration tests could look like with py.test instead of shell scripts.
For the proof of concept we migrate the integration tests for the list
subcommands and source builds.
It also spins up an Elasticsearch metrics store and checks that any
preconditions (basically Docker being present and up and running) are
met. Finally, it also integrates into CI by writing JUnit XML files for
test results.
Relates #736