Added New Test Definitions error, seekable, etc. Remove redundant test runs & Introduce dry test runs #507

indianwhocodes · 2019-04-16T21:03:32Z

Goals

Resolves testng encryptionCipher parameter does not appear to do anything #336 .
Eliminate redundant integration-test runs.
Test-Suite Size Reduction obtained from 609 tests -> 250 tests.
Class-Level Definition for every new integration-test that will be added in the future.
Incorporate dry run capability for the TestNG Suite of SDK to determine test-methods that are executed and the order in which they are run, without actually executing them.
These changes will further help in setting up a new Jenkins job java-manta-pre-it or simply update the existing one java-manta-it for new PR's made against the Java SDK.

Challenges

Unfortunately TestNG doesn't comprehensively support filtering a test suite by groups so we had to be more explicit about suite definition.

Overview

Adds a it.dryRun property that can be used to list the tests that would run with their respective TestNG params. TestNG unfortunately doesn't completely support filtering a test suite by groups so thorough detailing was required in suite definition.

Output of mvn verify it.dryRun=true using current testng-it.xml used with TestListingInterceptor:

[main] INFO  com.joyent.test.util.MantaPathSuiteListener [ ] - Base manta path for suite Java Manta SDK Integration Test Suite: /ashwin.nair/stor/java-manta-integration-tests/8669c9af-b512-4ac6-b7be-2390b6a5e61c/
[main] INFO  com.joyent.test.util.TestListingInterceptor [ ] - DRY-RUN: Listing [250] tests that would have run
--------------------------ALL-TESTS----------------------------------------------------------
main] INFO  com.joyent.test.util.MantaPathSuiteListener [ ] - Expected test count: 250
[main] WARN  com.joyent.test.util.MantaPathSuiteListener [ ] - Skipping suite cleanup since dry-run is enabled.
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.952 s - in TestSuite

List for the output for before & after -Dit.dryRun=true respectively . Note thatdry-run-after list is accurate since it shows single value for encryptionCipher i.e AES128/CTR/NoPadding. Other testing groups include cbc-cipher and gcm-cipher are also defined.

Changes

change usingEncryption (boolean) to just encryptionCipher (string). omission (null) disables encryption
add a new constructor for IntegrationTestConfigContext which handles the above
switch from @BeforeClass with @Parameters to test constructors with just @Parameters. Whenever SkipException was thrown or putDirectory was called in a test which needed @Parameters a minimal method was created for @BeforeClass. The goal here was to make it safe to instantiate test classes without incurring side-effects.
remove encryption param from tests which don't actually perform any object PUTs or MPU uploads (e.g. MantaClientDirectoriesIT)
change <packages> to <classes> and target specific classes.
document a new system property it.dryRun which can be used by the new interceptor. When this system property is truthy as defined by BooleanUtils#toBooleanObject, i.e. the following are all true: y, Y, t, T, on, ON, yes, YES, true, TRUE and all the mixed-case versions thereof
added a test listener called TestListingInterceptor which is used to collect suite definitions and disable actually running tests in the event it.dryrun is true.
defined new <test name></test> and the EXISTING ones in testing-it.xml more accurately using class-level specifications including TestNG Group Definitions : timeout, range-downloads, snaplinks, seekable, error, headers, directory, metadata, etc.
Following up on Fixing Issue 483 For Defining HeavyWeight Test Groups #492 which had to be closed due to dependency-vulnerabilities.
Changes are incorporated from the closed PR Remove redundant integration test runs #364 by Tomas which is to remove redundancies in integration-test runs.

cburroughs · 2019-04-17T18:19:48Z

Drive by color opinion: "nightly" isn't a term exclusive to Joyent, but still pretty far on the jargon scale. Is there a more self-descriptive term we could use? (slow? expensive?)

indianwhocodes · 2019-04-17T18:38:12Z

Drive by color opinion: "nightly" isn't a term exclusive to Joyent, but still pretty far on the jargon scale. Is there a more self-descriptive term we could use? (slow? expensive?)

Expensive Sounds Good.

cburroughs · 2019-04-22T19:46:38Z

I have not used testng groups before and am not sure if I understand how the flags are to be used. With -Dit.dryRun=true I got Expected test count: 232. If I added -Dgroups=headers I got Expected test count: 7, which seemed okay. But -DexcludeGroups=expensive and showed Expected test count: 232.

indianwhocodes · 2019-04-22T20:43:56Z

I have not used testng groups before and am not sure if I understand how the flags are to be used. With -Dit.dryRun=true I got Expected test count: 232. If I added -Dgroups=headers I got Expected test count: 7, which seemed okay. But -DexcludeGroups=expensive and showed Expected test count: 232.

Its because the expensive test group is enabled-false by default

dekobon · 2019-04-22T23:57:23Z

When running the test from Maven using mvn verify, I got the following errors:

[ERROR] Tests run: 221, Failures: 3, Errors: 0, Skipped: 25, Time elapsed: 2,381.243 s <<< FAILURE! - in TestSuite
[ERROR] beforeClass(com.joyent.manta.client.MantaClientMetadataIT)  Time elapsed: 0.019 s  <<< FAILURE!
java.lang.NullPointerException
	at com.joyent.manta.client.MantaClientMetadataIT.beforeClass(MantaClientMetadataIT.java:51)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:168)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:105)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[ERROR] beforeClass(com.joyent.manta.client.MantaClientMetadataIT)  Time elapsed: 4.03 s  <<< FAILURE!
java.lang.NullPointerException
	at com.joyent.manta.client.MantaClientMetadataIT.beforeClass(MantaClientMetadataIT.java:51)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:168)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:105)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[ERROR] beforeClass(com.joyent.manta.client.MantaClientMetadataIT)  Time elapsed: 9.977 s  <<< FAILURE!
java.lang.NullPointerException
	at com.joyent.manta.client.MantaClientMetadataIT.beforeClass(MantaClientMetadataIT.java:51)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:168)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:105)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR] com.joyent.manta.client.MantaClientMetadataIT.beforeClass(com.joyent.manta.client.MantaClientMetadataIT)
[ERROR]   Run 1: MantaClientMetadataIT.beforeClass:51 NullPointer
[INFO]   Run 2: PASS
[ERROR]   Run 3: MantaClientMetadataIT.beforeClass:51 NullPointer
[INFO]   Run 4: PASS
[ERROR]   Run 5: MantaClientMetadataIT.beforeClass:51 NullPointer
[INFO]   Run 6: PASS
[INFO] 
[INFO] 
[ERROR] Tests run: 205, Failures: 1, Errors: 0, Skipped: 11

java-manta-it/src/test/java/com/joyent/manta/client/MantaClientMetadataIT.java

...anta-it/src/test/java/com/joyent/manta/client/multipart/EncryptedJobsMultipartManagerIT.java

...t/src/test/java/com/joyent/manta/client/multipart/EncryptedServerSideMultipartManagerIT.java

...va/com/joyent/manta/client/multipart/EncryptedServerSideMultipartManagerSerializationIT.java

java-manta-it/src/test/java/com/joyent/manta/client/MantaClientMetadataIT.java

indianwhocodes · 2019-04-23T01:47:16Z

I have not used testng groups before and am not sure if I understand how the flags are to be used. With -Dit.dryRun=true I got Expected test count: 232. If I added -Dgroups=headers I got Expected test count: 7, which seemed okay. But -DexcludeGroups=expensive and showed Expected test count: 232.

Its because the expensive test group is enabled-false by default

I have not used testng groups before and am not sure if I understand how the flags are to be used. With -Dit.dryRun=true I got Expected test count: 232. If I added -Dgroups=headers I got Expected test count: 7, which seemed okay. But -DexcludeGroups=expensive and showed Expected test count: 232.

Also, I believe the comment here might be useful.

java-manta-it/src/test/resources/testng-it.xml

indianwhocodes · 2019-04-29T23:21:46Z

Deprecated classes coming within expensive testng-groups are being disabled by default

cburroughs · 2019-04-30T21:23:11Z

I have -- I think -- the latest branch checked out. Git tells me the last commit is 4d6d73b. Do I have that right?

When I do mvn verify -Dit.dryRun=true I get

[main] INFO  com.joyent.test.util.TestListingInterceptor [ ] - DRY-RUN: Listing [294] tests that would have run

But when I run mvn verify -Dit.dryRun=true -DexcludedGroups=expensive

[main] INFO  com.joyent.test.util.TestListingInterceptor [ ] - DRY-RUN: Listing [322] tests that would have run

Do these counts match yours? Shouldn't the number of expected tests go down when any -DexcludedGroups are given?

indianwhocodes · 2019-04-30T22:11:55Z

I have -- I think -- the latest branch checked out. Git tells me the last commit is 4d6d73b. Do I have that right?

When I do mvn verify -Dit.dryRun=true I get
[main] INFO  com.joyent.test.util.TestListingInterceptor [ ] - DRY-RUN: Listing [294] tests that would have run
But when I run mvn verify -Dit.dryRun=true -DexcludedGroups=expensive
[main] INFO  com.joyent.test.util.TestListingInterceptor [ ] - DRY-RUN: Listing [322] tests that would have run
Do these counts match yours? Shouldn't the number of expected tests go down when any -DexcludedGroups are given?

Yes, These counts match mine, but when you invoke -DexcludedGroups parameter from the command line, it overrides all the exclusion tags defined in the testng-it.xml file which consecutively results in leading to more redundant test runs. This PR is made so that we won't have to invoke -DexcludedGroups anymore and also eliminates redundant test runs since we can now source the testng-it (Integration-Test Suite) to perform the exclusions for us using the added <exclude> tags.

indianwhocodes · 2019-04-30T22:15:13Z

@cburroughs the discussion here will shed more light on what I am talking about. Also,

Unfortunately it looks like TestNG doesn't completely support filtering a test suite by groups so we need to be more explicit about suite definition.

cburroughs · 2019-05-01T13:50:22Z

Glad to hear I'm not crazy about the counts. Per our call, we have two use cases we want to be easy to from the command line (for jenkins). Anything fancier is okay as a convenience for developers and their IDEs:
(1) Run all of the tests we care about. This would be for nightly and after a commit to master.
(2) Run all of the tests we care about, except the really expensive ones. This would be for testing PRs.

With this PR the answer to (1) is just the default from running mvn verify. What is answer for (2)?

dekobon · 2019-05-21T16:24:15Z

I just noticed that EncryptedServerSideMultipartManagerIT doesn't have canListUploadsInProgress marked as @Test(groups = { "expensive" }).

Additionally, when running mvn clean verify it doesn't skip tests in the expensive group by default. Is that behavior by design?

indianwhocodes · 2019-06-07T21:25:18Z

When running mvn verify against the root directory.

[main] INFO  com.joyent.test.util.MantaPathSuiteListener [165.225.164.10 1b94e3c8-8967-11e9-b295-e336595f6b9e] - Removing base suite path: /ashwin.nair/stor/java-manta-integration-tests/$SUITE-UUID/
[INFO] Results:
[INFO] 
[WARNING] Tests run: 253, Failures: 0, Errors: 0, Skipped: 11



Total time: 24:37 min

indianwhocodes · 2019-06-07T21:27:40Z

If I run mvn verify it.dryRun=true as per the PR initial description I get
[ERROR] Unknown lifecycle phase "it.dryRun=true". You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-clean, clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1]
After a bit more reading, it seems you mean mvn verify -Dit.dryRun=true (it's correct further into your initial PR description)

All that remains is for me to ask why you opened this PR? What problem you solve, and how I can determine the problem has been solved (or feature added)?

Goals & Challenges updated in the PR Description along with additional changes in javadocs and testing documentation.

TESTING.md

cburroughs · 2019-06-28T01:10:11Z

Glad to hear I'm not crazy about the counts. Per our call, we have two use cases we want to be easy to from the command line (for jenkins). Anything fancier is okay as a convenience for developers and their IDEs:
(1) Run all of the tests we care about. This would be for nightly and after a commit to master.
(2) Run all of the tests we care about, except the really expensive ones. This would be for testing PRs.
With this PR the answer to (1) is just the default from running mvn verify. What is answer for (2)?

After discussing with Elijah, reached the conclusion that all expensive tests will be excluded from mvn verify which will be an answer to both aforementioned use-cases.

This essentially means for following use-cases:
1. execute `mvn clean verify -DincludedGroups=expensive`. if we are opting to test heavyweight operations involving `mpu` and `jobs`.

2. execute `mvn clean verify`

I'm afraid I'm still having trouble getting the two uses case to work, this gives me 250 tests:

$ mvn --projects java-manta-it --also-make  verify -Dit.dryRun=true | grep 'Expected test count'
[main] INFO  com.joyent.test.util.MantaPathSuiteListener [ ] - Expected test count: 250

But neither of these are giving me > 250 tests

 $ mvn --projects java-manta-it --also-make  verify -DincludedGroups=expensive -Dit.dryRun=true | grep 'Expected test count'
[main] INFO  com.joyent.test.util.MantaPathSuiteListener [ ] - Expected test count: 250
 $ mvn --projects java-manta-it --also-make verify -Dgroups=expensive -Dit.dryRun=true | grep 'Expected test count'
[main] INFO  com.joyent.test.util.MantaPathSuiteListener [ ] - Expected test count: 0

What is the set of flags to run all of the tests as we would for post-commit CI?

indianwhocodes · 2019-06-28T21:35:20Z

So we are entirely removing the approach of having an expensive test-group due to TestNG being extremely unreliable in its framework to support testng group exclusions cleanly as it was mentioned in #364 as well.

All integration-tests that have been anointed as expensive have been removed from the default Testing suite file so all we have to do now is to run mvn verify as a Jenkins job for all pending PR's since the default test-suite is already running a subset of all the integration-tests (excluding the unimportant:MantaClientJobIT, EncryptedJobsMultipartManager etc. and deprecated integration-tests).

indianwhocodes · 2019-06-28T21:37:30Z

Glad to hear I'm not crazy about the counts. Per our call, we have two use cases we want to be easy to from the command line (for jenkins). Anything fancier is okay as a convenience for developers and their IDEs:
(1) Run all of the tests we care about. This would be for nightly and after a commit to master.
(2) Run all of the tests we care about, except the really expensive ones. This would be for testing PRs.
With this PR the answer to (1) is just the default from running mvn verify. What is answer for (2)?

After discussing with Elijah, reached the conclusion that all expensive tests will be excluded from mvn verify which will be an answer to both aforementioned use-cases.
This essentially means for following use-cases:
1. execute `mvn clean verify -DincludedGroups=expensive`. if we are opting to test heavyweight operations involving `mpu` and `jobs`.

2. execute `mvn clean verify`
I'm afraid I'm still having trouble getting the two uses case to work, this gives me 250 tests:
$ mvn --projects java-manta-it --also-make  verify -Dit.dryRun=true | grep 'Expected test count'
[main] INFO  com.joyent.test.util.MantaPathSuiteListener [ ] - Expected test count: 250
But neither of these are giving me > 250 tests
 $ mvn --projects java-manta-it --also-make  verify -DincludedGroups=expensive -Dit.dryRun=true | grep 'Expected test count'
[main] INFO  com.joyent.test.util.MantaPathSuiteListener [ ] - Expected test count: 250
 $ mvn --projects java-manta-it --also-make verify -Dgroups=expensive -Dit.dryRun=true | grep 'Expected test count'
[main] INFO  com.joyent.test.util.MantaPathSuiteListener [ ] - Expected test count: 0
What is the set of flags to run all of the tests as we would for post-commit CI?

We would just be running these tests like we run normal integration-tests. For instance,
mvn verify -Dtest=MantaClientJobtIT

cburroughs · 2019-06-29T00:42:04Z

In that case, is it still necessary to switch from wildcards to a explicit list in the testng suite file? I thought we did that as part of trying to get groups to work, but it has an ongoing cost (future engineers will inevitably mess it up)

indianwhocodes · 2019-06-29T00:51:10Z

In that case, is it still necessary to switch from wildcards to a explicit list in the testng suite file? I thought we did that as part of trying to get groups to work, but it has an ongoing cost (future engineers will inevitably mess it up)

Yes but I explicitly added that in the documentation here that developers will have to add definitions to the testing suite once they come up with a new integration-test. I will make sure it gets added to the general SDK User Guide as well. Also, class-level definitions were in-place to reduce the redundancies significantly otherwise we were running each integration-test almost 6 times probably more.

cburroughs

I noted a few minor doc or formatting nits on the last read through. This is good to land once those are folded in. Thanks for shepherding this big set of changes through.

TESTING.md

java-manta-it/src/test/java/com/joyent/manta/config/IntegrationTestConfigContext.java

java-manta-it/src/test/java/com/joyent/test/util/TestListingInterceptor.java

indianwhocodes · 2019-07-01T18:35:05Z

I noted a few minor doc or formatting nits on the last read through. This is good to land once those are folded in. Thanks for shepherding this big set of changes through.

Thanks @cburroughs for this thorough review, learned a lot of much needed conventions, basics through this entire process.

dekobon · 2019-07-03T17:55:00Z

I ran the integration tests for the latest bits and found the following test failure:

[ERROR] Tests run: 256, Failures: 1, Errors: 0, Skipped: 14, Time elapsed: 953.694 s <<< FAILURE! - in TestSuite
[ERROR] pruneParentDirectoriesFull(com.joyent.manta.client.MantaClientDirectoriesIT)  Time elapsed: 29.197 s  <<< FAILURE!
java.lang.AssertionError: expected [false] but found [true]
        at org.testng.Assert.fail(Assert.java:96)
        at org.testng.Assert.failNotEquals(Assert.java:776)
        at org.testng.Assert.assertFalse(Assert.java:65)
        at org.testng.Assert.assertFalse(Assert.java:75)
        at com.joyent.manta.client.MantaClientDirectoriesIT.pruneParentDirectoriesFull(MantaClientDirectoriesIT.java:262)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
        at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
        at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
        at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989)
        at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
        at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

indianwhocodes · 2019-07-03T18:43:23Z

I ran the integration tests for the latest bits and found the following test failure:

[ERROR] Tests run: 256, Failures: 1, Errors: 0, Skipped: 14, Time elapsed: 953.694 s <<< FAILURE! - in TestSuite
[ERROR] pruneParentDirectoriesFull(com.joyent.manta.client.MantaClientDirectoriesIT)  Time elapsed: 29.197 s  <<< FAILURE!
java.lang.AssertionError: expected [false] but found [true]
        at org.testng.Assert.fail(Assert.java:96)
        at org.testng.Assert.failNotEquals(Assert.java:776)
        at org.testng.Assert.assertFalse(Assert.java:65)
        at org.testng.Assert.assertFalse(Assert.java:75)
        at com.joyent.manta.client.MantaClientDirectoriesIT.pruneParentDirectoriesFull(MantaClientDirectoriesIT.java:262)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
        at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
        at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
        at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989)
        at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
        at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

It seems this particular failure occurred due to the STRING_GENERATOR making some discrepancies when it came to creating the parentDir and childDir directories and some inconsistencies with their depths.
I ran this specific test separately and ran it while I invoked mvn verify and I didn't encounter any failures perhaps you could attach both String values so that I could interpret as to the reason it failed but the changes that have been checked in within #507 has nothing to do with this failure.

dekobon · 2019-07-03T18:44:53Z

I'm going to go ahead and approve the PR. Can you make another ticket to capture the problems with that particular test?

indianwhocodes · 2019-07-03T18:46:24Z

I'm going to go ahead and approve the PR. Can you make another ticket to capture the problems with that particular test?

Yes, Issue #542 created to actively track this issue.

indianwhocodes · 2019-07-03T21:34:24Z

This branch has been squashed and merged into master.

indianwhocodes mentioned this pull request Apr 16, 2019

Defining HeavyWeight and Other Test Groups #504

Closed

indianwhocodes added bug enhancement good first issue test-bug labels Apr 16, 2019

indianwhocodes self-assigned this Apr 16, 2019

indianwhocodes requested a review from dekobon April 16, 2019 21:09

indianwhocodes requested a review from cburroughs April 17, 2019 18:37

dekobon suggested changes Apr 22, 2019

View reviewed changes

indianwhocodes commented Apr 23, 2019

View reviewed changes

java-manta-it/src/test/java/com/joyent/manta/client/MantaClientMetadataIT.java Outdated Show resolved Hide resolved

indianwhocodes requested a review from dekobon April 23, 2019 01:43

dekobon suggested changes Apr 26, 2019

View reviewed changes

java-manta-it/src/test/resources/testng-it.xml Show resolved Hide resolved

indianwhocodes mentioned this pull request Apr 29, 2019

Create Test Groups that include Heavyweight Tests that are not run by default #483

Closed

Ashwin A Nair added 5 commits May 20, 2019 17:49

Added new testgroups

c0e58dd

Addressed Chris Review

fda4083

Address Elijah's Feedback

9094529

Final Changes

a7f9574

Remove <groups> tags

2e41a67

indianwhocodes requested a review from russelldb June 7, 2019 21:27

Merge branch 'master' into testgroups-branch

bf1e5ed

dekobon suggested changes Jun 12, 2019

View reviewed changes

TESTING.md Outdated Show resolved Hide resolved

TESTING.md Outdated Show resolved Hide resolved

Reinstated groups definitions, Docs Updated

2eb2a61

indianwhocodes requested a review from dekobon June 25, 2019 19:05

cburroughs approved these changes Jul 1, 2019

View reviewed changes

Chris suggestions on docs, whitespace correction

eec02cf

dekobon approved these changes Jul 3, 2019

View reviewed changes

indianwhocodes mentioned this pull request Jul 3, 2019

MantaClientDirectoriesIT.pruneParentDirectoriesFull fails at certain use-cases #542

Closed

indianwhocodes merged commit ed0f8e1 into TritonDataCenter:master Jul 3, 2019

indianwhocodes deleted the testgroups-branch branch July 3, 2019 21:33

indianwhocodes restored the testgroups-branch branch July 3, 2019 21:33

indianwhocodes deleted the testgroups-branch branch July 3, 2019 21:33

indianwhocodes restored the testgroups-branch branch July 3, 2019 21:33

indianwhocodes pushed a commit that referenced this pull request Jul 5, 2019

Reconfigure MantaClientIT to incorporate #507, #539

9b874b9

indianwhocodes mentioned this pull request Jul 11, 2019

Multiple test failures in MantaClientIT #545

Open

indianwhocodes deleted the testgroups-branch branch October 31, 2019 00:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added New Test Definitions error, seekable, etc. Remove redundant test runs & Introduce dry test runs #507

Added New Test Definitions error, seekable, etc. Remove redundant test runs & Introduce dry test runs #507

indianwhocodes commented Apr 16, 2019 •

edited

Loading

cburroughs commented Apr 17, 2019

indianwhocodes commented Apr 17, 2019 •

edited

Loading

cburroughs commented Apr 22, 2019

indianwhocodes commented Apr 22, 2019

dekobon commented Apr 22, 2019

indianwhocodes commented Apr 23, 2019

indianwhocodes commented Apr 29, 2019

cburroughs commented Apr 30, 2019

indianwhocodes commented Apr 30, 2019 •

edited

Loading

indianwhocodes commented Apr 30, 2019 •

edited

Loading

cburroughs commented May 1, 2019

dekobon commented May 21, 2019

indianwhocodes commented Jun 7, 2019

indianwhocodes commented Jun 7, 2019

cburroughs commented Jun 28, 2019

indianwhocodes commented Jun 28, 2019

indianwhocodes commented Jun 28, 2019

cburroughs commented Jun 29, 2019

indianwhocodes commented Jun 29, 2019 •

edited

Loading

cburroughs left a comment

indianwhocodes commented Jul 1, 2019

dekobon commented Jul 3, 2019

indianwhocodes commented Jul 3, 2019

dekobon commented Jul 3, 2019

indianwhocodes commented Jul 3, 2019 •

edited

Loading

indianwhocodes commented Jul 3, 2019

Added New Test Definitions error, seekable, etc. Remove redundant test runs & Introduce dry test runs #507

Added New Test Definitions error, seekable, etc. Remove redundant test runs & Introduce dry test runs #507

Conversation

indianwhocodes commented Apr 16, 2019 • edited Loading

Goals

Challenges

Overview

Changes

cburroughs commented Apr 17, 2019

indianwhocodes commented Apr 17, 2019 • edited Loading

cburroughs commented Apr 22, 2019

indianwhocodes commented Apr 22, 2019

dekobon commented Apr 22, 2019

indianwhocodes commented Apr 23, 2019

indianwhocodes commented Apr 29, 2019

cburroughs commented Apr 30, 2019

indianwhocodes commented Apr 30, 2019 • edited Loading

indianwhocodes commented Apr 30, 2019 • edited Loading

cburroughs commented May 1, 2019

dekobon commented May 21, 2019

indianwhocodes commented Jun 7, 2019

indianwhocodes commented Jun 7, 2019

cburroughs commented Jun 28, 2019

indianwhocodes commented Jun 28, 2019

indianwhocodes commented Jun 28, 2019

cburroughs commented Jun 29, 2019

indianwhocodes commented Jun 29, 2019 • edited Loading

cburroughs left a comment

Choose a reason for hiding this comment

indianwhocodes commented Jul 1, 2019

dekobon commented Jul 3, 2019

indianwhocodes commented Jul 3, 2019

dekobon commented Jul 3, 2019

indianwhocodes commented Jul 3, 2019 • edited Loading

indianwhocodes commented Jul 3, 2019

indianwhocodes commented Apr 16, 2019 •

edited

Loading

indianwhocodes commented Apr 17, 2019 •

edited

Loading

indianwhocodes commented Apr 30, 2019 •

edited

Loading

indianwhocodes commented Apr 30, 2019 •

edited

Loading

indianwhocodes commented Jun 29, 2019 •

edited

Loading

indianwhocodes commented Jul 3, 2019 •

edited

Loading