-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Grand unified theory of hooks #77
Conversation
Highlights:
Todo - test coverage - some unused code no doubt - implementation not 100% complete. Supplier<Widget> widget = let(() -> WidgetFactory.make());
describe("Some specs to run in parallel", with(parallelRunner(30), () -> {
it("will be run on one thread", () -> {
widget.get().doSomething(); // uses one widget even though apparently sharing `widget`
});
it("will be run on another thread", () -> {
widget.get().doSomething(); // uses another widget even though apparently sharing `widget` in parallel
});
})); |
* This is not the only way to achieve that - you can also build from {@link SupplyingHook} | ||
* but this captures the template for a complex hook. | ||
*/ | ||
public class AbstractSupplyingHook<T> extends Singleton<T> implements SupplyingHook<T> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would do both let
and junitTestClass
|
||
@Override | ||
public T get() { | ||
assertSpectrumInTestMode(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Borrowed from let
not a bad thing to have around.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And by borrowed, I mean stolen, given that let
is now a hook!
* Subclass of {@link Suite} that represent the fact that some tests are composed | ||
* of interrelated steps which add up to a single test. | ||
*/ | ||
final class CompositeTest extends Suite implements Atomic { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Extracted from #56 - this is necessary for the Atomic
to help us correctly.
/** | ||
* Allows the injection of suite configuration during test definition. A wrapper for the | ||
* suite which exposes configurables. | ||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Checkstyle forced me to write Javadoc as items stopped being internal when they switched packages.
* {@link SupplyingHook} or subclass {@link AbstractSupplyingHook}. | ||
*/ | ||
public interface Hook extends ThrowingConsumer<Block> { | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really want to modify aroundXXX
to use this rather than ThrowingConsumer
- we probably could, but for now I've left an adapter in place.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We haven't done a release with ThrowingConsumer
, so I wouldn't be too worried about protecting that API.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Throwing Consumer can die then.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ThrowingSupplier
on the other hand, was the item in question and I've removed it.. then restored it.
@@ -1,26 +0,0 @@ | |||
package com.greghaskins.spectrum; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not dead -> moved.
@@ -303,7 +296,7 @@ default T get() { | |||
* @param consumer to run each spec block | |||
*/ | |||
public static void aroundEach(ThrowingConsumer<com.greghaskins.spectrum.Block> consumer) { | |||
getCurrentSuiteBeingDeclared().aroundEach(consumer); | |||
getCurrentSuiteBeingDeclared().aroundEach(consumer::acceptOrThrow); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adapted!
throw new RuntimeException(checked); | ||
} | ||
} | ||
public interface ThrowingSupplier<T> extends com.greghaskins.spectrum.ThrowingSupplier<T> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Kept for posterity...
// the hooks - they will be turned into a chain of responsibility | ||
// so the first one will be executed last as the chain is built up | ||
// from first to last. | ||
private Hooks hooks = new Hooks(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So much smaller.
suite.beforeAll.addBlock(this.beforeAll); | ||
suite.beforeEach.addBlock(this.beforeEach); | ||
suite.afterEach.addBlock(this.afterEach); | ||
suite.aroundEach(this.aroundEach); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All this stuff now works with the hooks. The cascade from outer to inner is also rather nice - stopping us having to push it down at definition time.
} | ||
|
||
}).run(description, notifier); | ||
}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lots of code goes bye bye...
} | ||
|
||
private void runChildren(final RunNotifier notifier) { | ||
this.childRunner.runChildren(this, notifier); | ||
} | ||
|
||
private void runChild(final Child child, final RunNotifier notifier) { | ||
protected void runChild(final Child child, final RunNotifier notifier) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Visibility modified to enable the child to be run from the composite test subclass
Booyah! Coverage 100%! |
@@ -483,11 +484,10 @@ | |||
.map((failure) -> failure.getMessage()) | |||
.collect(Collectors.toList())); | |||
|
|||
it("report each error", () -> { | |||
assertThat(result.get().getFailureCount(), is(2)); | |||
it("report the error only once", () -> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: Let's not change this until we come to a decision on the other "to discuss" items.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Totally - was a temp test change. Let's revert it.
@@ -21,16 +21,20 @@ public void before() throws Exception { | |||
} | |||
|
|||
@Test | |||
public void thereAreTwoFailuresForEachAffectedTest() throws Exception { | |||
assertThat(this.result.getFailureCount(), is(4)); | |||
public void theActualTestsCanOnlyFailOnceEach() throws Exception { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: Let's not change this until we come to a decision on the other "to discuss" items.
@@ -10,17 +12,17 @@ | |||
/** | |||
* Represents the state of tagging for Spectrum - what it presently means. | |||
*/ | |||
class TaggingState { | |||
public class TaggingState { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this is getting promoted to the public API, then we might need to consider it's name. At first, I thought TaggingState
captured the tags for a spec, not the filter of which tags to run and which to ignore. Perhaps FilterCriteria
or something like that would be more appropriate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TaggingFilterCriteria
@@ -71,8 +71,7 @@ | |||
|
|||
assertThat(result.getFailureCount(), is(1)); | |||
assertThat(result.getFailures().get(0), is(failure("first spec", | |||
RuntimeException.class, "aroundEach did not run the block"))); | |||
|
|||
RuntimeException.class, "At least one of the test hooks did not run the test block."))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the generic hooks implementation, we certainly can't give as specific a message. Maybe we can at least include the location/trace to the offending hook?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you suggest how?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a minor one, we could address it in a future PR after #77 merges. I'm thinking we could capture the stacktrace at hook declaration time, and use that as the cause
when throwing an InvalidHookException
if it never called run()
. Also, I go back and forth on whether it's a good idea to allow hooks to call run()
multiple times -- it seems like doing so would probably break things. But I also don't want to limit the possibilities if we don't have to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Technically a hook could execute an it over multiple examples... so preventing multiple runs might be wrong...but this is not the easiest of frameworks to debug a failing test in. The simpler the hooks the better.
Let's defer all of this for post merge.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
import static com.greghaskins.spectrum.Spectrum.beforeEach; | ||
import static com.greghaskins.spectrum.Spectrum.configure; | ||
import static com.greghaskins.spectrum.Spectrum.describe; | ||
import static com.greghaskins.spectrum.Spectrum.it; | ||
import static com.greghaskins.spectrum.Spectrum.let; | ||
import static com.greghaskins.spectrum.internal.PreConditionBlock.with; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's discuss package structure and naming for PreConditions. If we are going to generalize the API to support things like parallelization, timeouts, etc. then we might want a different term. It feels kind of like some sort of Decorator
maybe?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Decorator is not quite it. I'll try a rename to BlockConfiguration
* destroyed. This is a twist on the usual singleton pattern, in that normally the | ||
* singleton stays around indefinitely. This version is also thread safe. | ||
*/ | ||
public class Singleton<T> implements Supplier<T> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ashleyfrieze I think I understand how this and AbstractSupplyingHook
work to provide better thread safety, and speed up parallelism. Let me share my understanding and you tell me if it's correct:
- With different threads running the same hooks, it could lead to weird cross-thread behavior if one thread steps on the state of another thread.
- A traditional approach might use
synchronized
blocks to mitigate that, but they are slow and limit parallelism since threads would have to wait for others to finish before entering. Also deadlocking is a thing that sucks. - In most cases, we actually don't want shared state across threads, so traditional synchronizing makes even less sense. It's much easier to reason about the code as if it was single-threaded.
- A
Singleton<T>
is kind of like aVariable<T>
that allows each thread to have its own, separate value of the variable. - This means that, for example, setup and teardown happening on one thread probably won't collide as much with tests that are running on another thread.
- Specifically, this will help with variables and state created and managed by the specs themselves. It doesn't solve anything related to side-effects such as databases or other external resources.
- The
Singleton
itself makes no attempt to limit changing, resetting, or re-creating its values, unlike a traditional singleton which is constructed once and shared.
Regardless, it took me a while to form this understanding, which might mean we need to revisit naming or structure of these classes. Maybe ThreadSingleton
or ConcurrentVariable
? Come to mention it, what is the real difference between this Singleton
and the built-in ThreadLocal
?
In doing some quick research, I found the commentary in this StackOverflow thread interesting. Oh, boy, threading is fun.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, and another thought/idea. We might try to explore the possible thread-safety failure points of Spectrum by writing some tests. When I have to rely too much on keeping the right model in my head and thinking things through properly, I start getting into trouble. That's why I love my TDD. Of course, threading-related automated tests are notoriously tricky to write. sigh.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My aim is to make it really thread safe in #61 for now, I'm just laying foundations.
I'm going to change the singleton stuff. You're right. It's got surprise in it. I'm going to claim that I am right to have chosen the name, since a Singleton is actually capable of abstracting the correct creation of an object and having more than one if you're doing things across threads.
In fact, though. The Variable
needs to be threadsafe, and it seems like let
and Variable
ought to share more implementation. Watch this space.
set(before()); | ||
block.run(); | ||
after(); | ||
clear(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At a minimum, clear()
should be in a finally
block. Depending on the outcome of other discussions, after()
may also need to be.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
@@ -188,7 +205,8 @@ public static Configuration configure() { | |||
* @param block {@link com.greghaskins.spectrum.Block} to run once before each spec | |||
*/ | |||
public static void beforeEach(final com.greghaskins.spectrum.Block block) { | |||
getCurrentSuiteBeingDeclared().beforeEach(block); | |||
addHook(new HookContext(before(block), getDepth(), | |||
HookContext.AppliesTo.ATOMIC_ONLY, HookContext.Precedence.LOCAL)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
all hookcontexts created by Spectrum
which finds it is to provide the depth.
The Precedence names and application are arbitrary and "whatever I had to do to get the tests to work"
|
||
private static final Deque<Suite> suiteStack = new ArrayDeque<>(); | ||
private static final Variable<Deque<Suite>> suiteStack = new Variable<>(ArrayDeque::new); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a threadsafe Deque
- I guess this is itself just a wrapper for ThreadLocal
but it feels nice using a Spectrum construct.
final com.greghaskins.spectrum.Block definitionBlock) { | ||
suiteStack.push(suite); | ||
getSuiteStack().push(suite); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Abstracted out our own field now it's a supplier. variable.get().push()
felt too long.
private Set<String> namesUsed = new HashSet<>(); | ||
private final TaggingFilterCriteria tagging; | ||
private BlockConfiguration preconditions = BlockConfiguration.Factory.defaultPreConditions(); | ||
private NameSanitiser nameSanitiser = new NameSanitiser(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hid the name sanitiser's implementation in another class to reduce Suite
.
@@ -42,7 +43,8 @@ | |||
} | |||
|
|||
static Suite rootSuite(final Description description) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rootSuite
was a song from Chitty Chitty Bang Bang.
@@ -27,7 +27,8 @@ public void thereIsOneFailureForEachAffectedTest() throws Exception { | |||
@Test | |||
public void theFailureExplainsWhatHappened() throws Exception { | |||
assertThat(this.result.getFailures().get(0), | |||
is(failure("a passing test", Fixture.SomeException.class, "kaboom"))); | |||
is(failure("a passing test", RuntimeException.class, | |||
"given.a.spec.with.exception.in.aftereach.block.Fixture$SomeException: kaboom"))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed purely down to the checked exception wrapping that now happens in ThrowingConsumer
@@ -13,11 +13,11 @@ | |||
describe("an exploding beforeEach", () -> { | |||
|
|||
beforeEach(() -> { | |||
throw new SomeException("kaboom"); | |||
throw new SomeException("beforeEach went kaboom"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just made the message more diagnostic.
@@ -34,7 +34,7 @@ | |||
return Spec.class; | |||
} | |||
|
|||
public static class SomeException extends Exception { | |||
public static class SomeException extends RuntimeException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed to unchecked to preserve original message.
@@ -28,9 +28,11 @@ public void thereAreTwoFailuresForEachAffectedTest() throws Exception { | |||
@Test | |||
public void theFailuresExplainWhatHappened() throws Exception { | |||
assertThat(this.result.getFailures().get(0), | |||
is(failure("a failing test", Fixture.SomeException.class, "kaboom"))); | |||
is(failure("a failing test", Fixture.SomeException.class, | |||
"beforeEach went kaboom"))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here's where the test uses the more specific error message - no tricks!
assertThat(numbers, contains(1, 2, 3, 4)); | ||
}); | ||
|
||
it("runs afterEach blocks from parent context", () -> { | ||
assertThat(numbers, contains(1, 2, 3, 4)); | ||
// if afterEach block didn't call clear, then | ||
// the numbers would be of size 8, not a fresh four |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added narrative to explain what the test means.
@@ -486,7 +490,7 @@ | |||
it("report each error", () -> { | |||
assertThat(result.get().getFailureCount(), is(2)); | |||
|
|||
assertThat(failureMessages.get(), hasItem("boom 1")); | |||
assertThat(failureMessages.get(), hasItem("java.lang.Exception: boom 1")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The checked exception gets wrapped. This only happens in the innermost consumer. Outer layers are (and can be) more permissive of unchecked exceptions.
@@ -502,12 +506,12 @@ | |||
(result) -> () -> result.get() | |||
.getFailures() | |||
.stream() | |||
.map((failure) -> failure.getMessage()) | |||
.map(Failure::getMessage) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Function reference beats lambda.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! I'm still getting used to all these cool Java 8 features.
.collect(Collectors.toList()); | ||
|
||
describe("in beforeEach and afterEach", () -> { | ||
|
||
final Supplier<List<String>> exceptionsThrown = let(() -> new ArrayList<>()); | ||
final Supplier<List<String>> exceptionsThrown = let(ArrayList::new); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Function reference again.
@@ -519,21 +523,21 @@ | |||
describe("an explosive suite", () -> { | |||
|
|||
beforeEach(() -> { | |||
throw recordException.apply(new Exception("boom beforeEach 1")); | |||
throw recordException.apply(new RuntimeException("boom beforeEach 1")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Runtime exceptions make the tests easier to read/write.
@ashleyfrieze Most excellent! A very nice overhaul of the codebase, with all original tests intact and passing! I'm prepared to merge this PR as-is, with a few to-dos on my list before the next release. I think it makes sense to resolve those as separate PRs, since this one is big enough! With all the features we've been adding, it feels like we're approaching a
The above bits will probably affect this code, perhaps moving pieces around and other minor changes. If you're good with that, then I will hit the Merge button on #77. |
Yeah. Merge merge merge! I can redo my JUnit rules piece then. Awesome! |
…resent, and supporting mix-in objects applied in the hierarchy as a special type of supplier. Updated the Hooks mechanism from greghaskins#77 to allow the passing of RunNotifier and Description around, and hooked in a private version of the rules handling code from JUnit which adapts from Hooks to Statement objects. Demonstrated the whole thing with Spring, Mockito and JUnit native rules, and managed to reverse some of the changes to exception handling brought by greghaskins#77
…resent, and supporting mix-in objects applied in the hierarchy as a special type of supplier. Updated the Hooks mechanism from greghaskins#77 to allow the passing of RunNotifier and Description around, and hooked in a private version of the rules handling code from JUnit which adapts from Hooks to Statement objects. Demonstrated the whole thing with Spring, Mockito and JUnit native rules, and managed to reverse some of the changes to exception handling brought by greghaskins#77
…present, and supporting mix-in objects applied in the hierarchy as a special type of supplier. Updated the Hooks mechanism from greghaskins#77 to allow the passing of RunNotifier and Description around, and hooked in a private version of the rules handling code from JUnit which adapts from Hooks to Statement objects. Demonstrated the whole thing with Spring, Mockito and JUnit native rules, and managed to reverse some of the changes to exception handling brought by greghaskins#77
Still WIP at the moment - demonstrated purely by a replacement of the
let
function.I've reorganised the package structure, and I'm in the process of killing off all arounds/befores etc. First target - the humble
let
.Code coverage will not be 100% yet as there are unused features.
Intended as a fix for #76 and #73. Eventually. Intended as a new platform on which to build #56