Test Framework - Hamcrest to AssertJ

[DanVanAtta]

I wouldn't mind seeing new test cases use 'AssertJ'.

Hamcrest is showing some age

Switching test framework is not a priority per-say, but I think there is probably a greenlight to stop using Hamcrest in favor of AssertJ. A rule of thumb would be any given test file shall not mix both Hamcrest & AssertJ.

I'm a bit concerned about a lava-flow problem where we wind up with two different test-assertions for a very long time and then perhaps introduce a third. If we start introducing assertJ, IMO we should stick with assertJ until all of Hamcrest is gone.

Any thoughts one way or another?

[asvitkine]

Is there some summary of what AssertJ brings?

My main gripes with hamcrest are:

1. Some assertion names are very unintuitive and I guess it's so baked in that it will never change. (e.g. things like https://stackoverflow.com/questions/33840531/how-do-hamcrests-hasitems-contains-and-containsinanyorder-differ, but there's a bunch)
2. A bit related to the above, but needing to look up the right thing to use because it's not always obvious
3. Mostly an IntelliJ issue, but sometimes it not suggesting something from hamcrest for the import at all, even when such a thing exists, instead suggesting some other name-collided symbols from other libraries.
4. Things being a bit verbose

Sometimes I wonder if just sticking to the simple things JUnit has is better (and perhaps augmenting those with our own implemented asserts that we can have a library of).

I'm also not in love with N different testing frameworks though. So adding a third doesn't seem great either, unless we can do some automated mass migration.

[DanVanAtta]

assertJ resources

The quickstart guide on assertJ is a decent resource:

• https://joel-costigliola.github.io/assertj/assertj-core-quick-start.html
• https://joel-costigliola.github.io/assertj/assertj-core-features-highlight.html

What choices do we have?

We have a bit of a choice before us:

• (A) we can start writing new tests in Junit, in which case we have "n" frameworks (n=2)
• (B) we can keep writing everything in hamcrest, even with its detriments
• (C) we can start writing new tests in AssertJ thereby having two frameworks
• (D) we can try to mass migrate all tests to AssertJ

I don't think there is a decent mass migration tool out there, and even then, I'm not sure if it's worth the time/effort.

What do we need from any testing framework?

To inform this conversation further, it's very important to consider the scaling cost of tests & their overhead. Consider the case of breaking 100 tests, it is a world of difference if that takes 5 seconds to debug each one, or it takes one minute per test. It is thus imperative to as quickly as possible identify the following three for any given test:

• what exactly is it testing
• why is it a valid test to begin with (why do we expect the asserted conditions to be true at all?)
• exactly why has it failed

Problems with Junit

Junit suffers from a number of issues:

• bad/incomplete error messages
• bad syntax
  • Specifically assertEquals flips the expected vs actual that would be natural for Java programmers. Junit assertEquals uses a C style pattern that was used to avoid null pointers. EG: "if( a == 20 )" vs "if( 20 == a)", the latter is very unnatural looking for a java programmer, yet that is the style assertEquals uses.
• not much support for common test assertions
  • doing things like asserting a list contains an item is somewhat painful, let alone printing a good debug statement

Hamcrest helps with all of those issues. To your list of issues with Hamcrest @asvitkine , I would emphasize the static import hell. At some point a person learns the library pretty well and starts to know where the common comparators live. At that point it is easy to work with, but I can't help but think this is a shining example of "barrier to entry".

Why AssertJ

In short, it solves the problems of Junit that Hamcrest solves. Second, it uses auto-complete like crazy - making it very easy to work with.

Example

The below code snippet of asserting a 'BigDecimal' value is less than 20, I think is very illustrative (comments below each assertion are the test output for each):

import org.hamcrest.MatcherAssert;
import org.junit.jupiter.api.Test;
import java.math.BigDecimal;
import static org.hamcrest.core.Is.is;
import static org.hamcrest.number.OrderingComparison.lessThan;

public class Example {

  @Test
  void testCase_A__bigDecimalLessThan20__withJunit() {
    BigDecimal actual = new BigDecimal(100);
    org.junit.jupiter.api.Assertions.assertTrue(actual.compareTo(BigDecimal.valueOf(20)) < 0);
    /*
    Expected :true
    Actual   :false
     */
  }

  @Test
  void testCase_B__bigDecimalLessThan20__withJunitVariant2() {
    BigDecimal actual = new BigDecimal(100);
    org.junit.jupiter.api.Assertions.assertTrue(
        actual.compareTo(BigDecimal.valueOf(20)) < 0,
        () -> String.format("Actual: %s was expected to be less than 20", actual));
    /*
    Actual: 100 was expected to be less than 20
    Expected :true
    Actual   :false
     */
  }


  @Test
  void testCase_C__bigDecimalLessThan20__withHamcrest() {
    BigDecimal actual = new BigDecimal(100);
    MatcherAssert.assertThat(actual, is(lessThan(BigDecimal.valueOf(20))));
    /*
    Expected: is a value less than <20>
    but: <100> was greater than <20>
     */
  }


  @Test
  void testCase_D__bigDecimalLessThan20__withAssertJ() {
    BigDecimal actual = new BigDecimal(100);
    org.assertj.core.api.Assertions.assertThat(actual).isLessThan(BigDecimal.valueOf(20));
    /*
    Expecting:
     <100>
    to be less than:
     <20>
     */
  }

}

[asvitkine]

Thanks for the example. I do like the AssertJ syntax better than hamcrest (as well as over JUnit).

If it also has more intuitive names for various tests you can do on collections, I'm sold.

[DanVanAtta]

Thanks for being open @asvitkine . I think I should have mentioned that in short assertJ is effectively just better than hamcrest (IMO), while Junit still has some real shortcomings.

[asvitkine]

To close the loop on this, I added our first use of AssertJ in 14b6ca4.