-
Notifications
You must be signed in to change notification settings - Fork 472
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multi-OS: Provide binaries #585
Multi-OS: Provide binaries #585
Conversation
… simplify makefile and maybe cross-platform differences
.travis.yml
Outdated
install: | ||
- go get -u golang.org/x/lint/golint | ||
- curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter | ||
- chmod +x ./cc-test-reporter | ||
- ./ci/install.sh "${TRAVIS_OS_NAME}" | ||
|
||
before_script: | ||
- ./cc-test-reporter before-build | ||
- ./ci/before-build.sh "${TRAVIS_OS_NAME}" | ||
|
||
script: | ||
- make all | ||
- ./ci/build.sh "${TRAVIS_OS_NAME}" | ||
|
||
after_script: | ||
- ./cc-test-reporter after-build --exit-code $TRAVIS_TEST_RESULT -d | ||
- ./ci/after-build.sh "${TRAVIS_OS_NAME}" | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've factored these commands into scripts where it's cleaner to account for the cross-platform differences than within the CI configuration.
|
||
test-all: fmt lint vet test test-int-all |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I introduced test-short-all
to represent all the short tests like lints and unit-tests, which can run on each platform.
The integration tests can't run on non-linux unless we come up with a different approach, but I definitely didn't want to include that scope here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Honestly, I give up on trying to have a proper integration test for Windows/OSX
Solid unit tests + a small smoke test might be sufficient.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, 👍. If we think it becomes worthwhile to do more than unit-test, I reckon it'll be reasonably straightforward to achieve via TestMain
and blackbox'ing against some golden files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curious on your Test mail suggestion. Whenever you have a chance (no rush). Can you link me a psudocode gist example of what you mean.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://github.com/aelsabbahy/goss/pull/407/files#diff-2e387891f2d94197d8fc70573226de67R17-R34 - the job of the TestMain
is to make sure that the binary is present; in this example it's done via go build
, but it doesn't matter what the implementation is.
https://github.com/aelsabbahy/goss/pull/407/files#diff-2e387891f2d94197d8fc70573226de67R1 - use a build-tag so that these tests only run via go test -tags integration
- this would allow the tests to only be run after the build automation has had a chance to build the release binaries, at which point TestMain
could just choose which one to use based on runtime.GOOS
/runtime.GOARCH
.
https://github.com/aelsabbahy/goss/pull/407/files#diff-2e387891f2d94197d8fc70573226de67R13 - rendon/testcli
is a package that looked the most similar to a setup I'm familiar with from another language's ecosystem. It makes a small wrapper around os.exec and then allows one to assert against stdout/stderr.
https://github.com/aelsabbahy/goss/pull/407/files#diff-2e387891f2d94197d8fc70573226de67R36-R63 - table-driven tests that run the binary and compare the output to some golden files, with the option to update those.
|
||
install: release/goss-linux-amd64 | ||
$(info INFO: Starting build $@) | ||
cp release/$(cmd)-linux-amd64 $(GOPATH)/bin/goss | ||
|
||
test: | ||
$(info INFO: Starting build $@) | ||
{ \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similar to inside travis, factored these into scripts where it becomes easier to deal with the cross-platform differences.
docs/platform-feature-parity.md
Outdated
|
||
| Feature | `linux` implementation + unit-tests | `linux` integration-tests | `macOS` implementation + unit-tests | `macOS` integration-tests | `Windows` implementation + unit-tests | `Windows` integration-tests | | ||
|:---------------|:--------|:-----------------:|:----------------:|:-----------------:|:----------------:|:-------------------:|:------------------:| | ||
| Assertion: `addr` | √ | √ | | | | | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can format this via a tool, but honestly I think that will just make it more fiddly to edit; tables in markdown are fiddly, so I've done the bare minimum.
I don't think this is complete, but it's illustrative. What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't render correctly for me. Maybe a typo somewhere?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm.. does Linux really have unit tests?
Maybe the columns need to be:
Feature, linux, windows, osx
And symbols for fully support, partial support, and no support.
Maybe for partial support, explain the subset of attributes that don't work.
Thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Columns sounds good to me. It works conceptually, but I think will be hard to deal with inside the table formatting. I'll make headings under the table and page links from the cells where more detail is needed. What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How's this? It's now basically a full test-plan, ready to be filled out. I don't intend to fill that out inside this PR more than what I've done to illustrate how to do it, though; sound ok?
I've expanded the description inside #586 since that existed already. If you prefer, I can transfer the bulk of that to an issue? I was partly responding to your comment in this PR, partly to #586 (comment). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Couple of comments.. and a question.
Question, why do the travis-ci for windows and osx differ in printing alpha warning?
|
||
#### Windows `add` | ||
|
||
```powershell |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
curious.. why does travis-CI not show this message?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The goss log.
output goes to stderr
and (apparently, I didn't know this until just now) travis doesn't capture stderr
.
You can try this out locally:
TRAVIS_OS_NAME="windows "ci/build.sh "windows" >stdout.txt
and you will see only the stderr
output since stdout
will now appear only within stdout.txt
.
Here is the output I get:
Pete@monsterbox ~/src/improbable/goss (multi-os-binaries)
λ TRAVIS_OS_NAME=windows ci/build.sh windows > stdout.txt
/usr/bin/sh: golint: command not found
2020/06/10 10:16:42 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:42 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:42 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:42 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:42 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:42 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:43 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:43 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:43 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:43 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:43 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:43 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:43 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:43 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:43 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:43 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:43 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
2020/06/10 10:16:43 WARNING: goss for this platform ("Windows") is alpha-quality, work-in-progress, and not yet exercised within continuous integration.
You should not expect everything to work. Treat linux as the canonical behaviour to expect.
Please see https://github.com/aelsabbahy/goss/tree/master/docs/platform-feature-parity.md to set your expectations and see progress.
Please file issues via https://github.com/aelsabbahy/goss/issues/new/choose
Pull requests and bug reports very welcome.
system/file_windows.go
Outdated
} | ||
|
||
func (f *DefFile) Owner() (string, error) { | ||
return getUserForUid(1000) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no data is better than wrong data? What does the hardcoded 1000 do here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd rather return a blank or not-implemented.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in 143e895
My plan is to merge this next week and released with a couple of other PRs. Didn't get any feedback from those tagged.. so.. might as well just get this wrapped up and released. |
@petemounce This is pretty much queued for the next release, just waiting on the two points above. |
Awesome. |
Pulled the code down locally to give it a quick sanity check before merging.
Seems this PR is a little too aggressive in what it requires travis_ci for. I should be able to continue to run |
Definitely. I'll fix that now. |
@aelsabbahy I don't understand why currently, but one linux http integration test is currently failing - goss-shared.yaml:L191-198. When I curl that address I get a 200 OK in a tiny amount of time (I've taken out the schannel parts as noise):
Whereas the integration test goss output is
I'm somewhat at a loss - that looks like it should be passing to me? |
Ugh.. I'll look at this tomorrow, flakey tests annoy me. Really need to switch to a completely controlled test environment and just drop the external dependencies. This could be fun for unit/integration tests on the to level. |
Oh, also -
The integration tests fail on Windows because git-bash doesn't come with |
That looks nice. re: environment - we could lean on a docker-compose file to stand up local DNS/http/etc endpoints to aim at, at some point? I hate flaky tests too. |
Agree. I go back and forth in my head if I want to be more unit test heavy or not. By the way, restarting the build got it green. |
I think flakes are insidiously bad, so I'm inclined to own remote dependencies where possible.
🤕 |
I just noticed that the build dependency on GO_FILES is missing, when I switch branches and do a This isn't something I'll hold the PR over.. but curious if it was moved intentionally or just missed with the Makefile refactor. If not intentional, I can create a quick change adding it back in. |
Thank you so much for this. A lot of time and effort went into this work and it is greatly appreciated! |
My pleasure! |
I think that was inadvertent. Makefiles are not a core skill of mine - I thought it was unused but missed that it was being used as a dependency to invalidate builds to make rebuilds happen. In one of the earlier commits I found a way to do it cross platform with |
Fixes #551, #405, #406, #403, #448, #385, #380, #298 (or supercedes them)
Checklist
make test-all
(UNIX) passes. CI will also test thisDescription of change
Adds binaries for macOS and Windows, plus feature-parity matrix doc / test-plan.