Tests seem to always result in PASS #8
Comments
|
@Travisivart What version of gdt-dev/kube do you have in your |
Travisivart
changed the title
|
github.com/gdt-dev/gdt v1.1.0 |
|
@Travisivart same for gdt-dev/kube? |
|
Yes, same for both of them. |
|
OK, I think I've figured out at least part of the reason this is happening. If I change the kube repository's - name: deployment-matches-expected-fields
timeout:
after: 20s
kube:
get: deployments/nginx
assert:
matches:
spec:
replicas: 2
template:
metadata:
labels:
app: nginx
status:
readyReplicas: 2to this: - name: deployment-matches-expected-fields
timeout:
after: 20s
kube:
get: deployments/nginx
assert:
matches:
spec:
replicas: 2
template:
metadata:
labels:
app: nginx
status:
eadyReplicas: 2(Note misspelling of "readyReplicas" as "eadyReplicas") I get the following results when running Note, the overall test scenario (matches.yaml) produces a FAIL (as it should), however the failure is from the timeout exceeded instead of the assertion failure that last occurred before the timeout fired. And, worse, the test unit (deployment-matches-expected-fields) returns PASS where it should return FAIL. |
|
Yeah, if I run so this is an issue of the gdt-kube |
This patch addresses a couple related problems, all with the evaluation of testing.T failures. When `testing.T.Run()` is executed, a new goroutine is spawned with a new `testing.T` pointer. This specific goroutine's `testing.T` pointer needs to have *its* `testing.T.Error()` method called in order for that sub-test to be marked failed. We were erroneously calling `testing.T.Error()` within the `Scenario.Run()` method instead of inside the `Spec.Eval()` method, which resulted in the test scenario being marked failed instead of the individual test unit. We address the exec plugin's `Spec.Eval()` in this patch to call `testing.T.Error()` on any assertion failure however additional patches are coming for the http and kube plugins. Finally, I made a change to the `gdterrors.TimeoutExceeded()` function to allow for an assertion failure message to be supplied to the error producer, making it easier for folks to see "this test assertion failed to succeed before a timeout of (duration)". Addresses Issue #8 Signed-off-by: Jay Pipes <jaypipes@gmail.com>
Brings in gdt@v1.1.1 and ensures that the test units/specs call `testing.T.Error()` instead of relying on the `Scenario.Run()` to do that. Also adds a custom YAML unmarshaler for the `Expect` struct and adds better parse-time errors for the `matches` field as requested in Issue 8. Addresses Issue gdt-dev/gdt#8 Signed-off-by: Jay Pipes <jaypipes@gmail.com>
|
OK @Travisivart I've cut |
|
This was fixed in gdt 1.1.1 |
I have a custom resource (csn-test1) which is already defined. I am trying to verify that it has a provisioningStatus of Succeeded. The test yaml file is defined as:
It seems like it doesn't matter what the 'status: Succeeded' field is set to, it will always result in a 'PASS' on the test, instead of the expected FAIL. I tried setting the field to 'status: Succ' which should FAIL but the test case still ends in PASS.
In fact, I was able to drop the entire 'matches' so it doesn't have anything to match and it still resulted in PASS. I would have expected it to fail if it has nothing to match on, or at least some sort of compile error if it has an assertion to make but nothing is actually defined for it.
Maybe something like this:
=== RUN TestNetwork
suite_test.go:18: invalid YAML: expected map field at line 9, column 11
--- FAIL: TestNetwork (0.00s)
FAIL
FAIL test/functional-gdt 0.016s
FAIL
Which you get if you define assert with nothing under it like so:
The text was updated successfully, but these errors were encountered: