JUnit 5 – Dynamic Tests

With JUnit 5’s dynamic tests it is possible to define full fledged test cases at run time. With it, tests can be created from parameters, external data source, or simple lambda expressions. This fixes a tremendous weakness of JUnit 4, where tests had to be defined at compile time.

Overview

Other posts in this series about JUnit 5:

(If you'd rather hear me talk about JUnit 5, check out the recent vJUG session or my presentation at JavaZone.)

This series is based on the pre-release version Milestone 3 and will get updated when a new milestone or the general availability release gets published.

Most of what you will read here and more can be found in the emerging JUnit 5 user guide (that link went to the Milestone 3 version - you can find the most current version here). The code samples I show here can be found on GitHub.

Static Tests

JUnit 3 identified tests by parsing method names and checking whether they started with test. JUnit 4 took advantage of the (then new) annotations and introduced @Test, which gave us much more freedom. Both of these techniques share the same approach: Tests are defined at compile time.

This can turn out to be quite limiting, though. Consider, for example, the common scenario that the same test is supposed to be executed for a variety of input data, in this case for many different points:

What are our options? The most straight forward is to create a number of interesting points then just call our test method in a loop:

If we do that, though, JUnit will see our loop as a single tests. This means that tests are only executed until the first fails, reporting will suffer, and tool support is generally subpar.

There are a couple of JUnit 4 features and extensions that address this issue. They all more or less work but are often limited to a specific use case (Theories), are awkward to use (Parameterized), and usually require a runner (like the commendable JUnitParams). The reason is that they all suffer from the same limitation: JUnit 4 does not really support creating tests at run time.

The same applies to creating tests with lambdas. Some would like to define tests like this:

This is of course just an ideal – it does not even compile in Java. Nevertheless, it would be interesting to see how close we can get. Alas, individual lambdas can not be statically identified, either, so the same limitation applies here.

But I wouldn’t be writing all of this if JUnit 5 did not propose a solution: Dynamic tests to the rescue!

Dynamic Tests

Since very recently the JUnit 5 code base sports a new type as well as a new annotation and together they address our problem.

First, there is DynamicTest, a simple wrapper for a test. It has a name and holds the code that makes up the test’s body. The latter happens in the form of an Executable, which is like a Runnable but can throw any Throwable (formidable naming). It is created with a static factory method:

Then there is @TestFactory, which can annotate methods. Those methods must return an Iterator, Iterable, or Stream of dynamic tests. (This can of course not be enforced at compile time so JUnit will barf at run time if we return something else.)

It is easy to see how they cooperate:

  1. When looking for @Test methods, JUnit will also discover @TestFactory methods.
  2. While building the test tree, it will execute these methods and add the generated tests to the tree.
  3. Eventually, the tests will be executed.

We are hence able to dynamically create tests at run time:

Let’s see how we can use it to solve the problems we described above.

Parameterized Tests

To create parameterized tests, we do something very similar to before:

The critical difference to what we did above is that we do not directly execute testDistanceComputation anymore. Instead we create a dynamic test for each datum, which means that JUnit will know that these are many tests and not just one.

In cases like this we might use a different method to generate the dynamic tests:

Here we hand our test data to stream and then tell it how to create names and tests from that.

So what do you think? Maybe something along the lines of “It’s cool that JUnit 5 treats these as individual tests but syntactically it’s still cumbersome”? Well, at least that’s what I think. The feature is nice but somewhat ungainly.

But this is only Milestone 3 so there is enough time for improvement. Maybe extensions can provide a more comfortable way to create dynamic tests but I don’t quite see how. I guess, a new extension point would help.

Lambda Tests

Ok, let’s see how close we can get to the much-coveted lambda tests. Now, dynamic tests were not explicitly created for this so we have to tinker a bit. (This tinkering is, err, “heavily inspired” by one of Jens Schauder‘s presentations about JUnit 5. Thanks Jens!)

A dynamic test needs a name and an executable and it sounds reasonable to create the latter with a lambda. To be able to do do this, though, we need a target, i.e. something the lambda is assigned to. A method parameter comes to mind…

But what would that method do? Obviously it should create a dynamic test but then what? Maybe we can dump that test somewhere and have JUnit pick it up later?

Ok, that looks promising. But where do we get an instance of LambdaTest? The easiest solution would be for our test class to simply extend it and then repeatedly call registerTest. If we do so, we might prefer a shorter name, though; and we can also make it protected:

Looks like we’re getting there. All that’s left is to call λ and the only apparent way to do this is from inside our test class’ constructor:

We’re done tinkering. To get further, we have to start hacking. Ever heard of double brace initialization? This is a somewhat strange feature that uses an initializer block to execute code during construction. (I used to think that this creates an anonymous subclass, but that’s not the case in this scenario as Duncan and, in more detail, Reinhard pointed out.) With it, we can go further:

If we’re really eager we can shave off another couple of symbols. With this one weird trick (we’re now being inspired by Benji Weber), we can determine a lambda’s parameter name via reflection and use that as the test’s name. To take advantage of that we need a new interface and have to change LambdaTest::λ a bit:

Putting it all together we can create tests as follows:

What do you think? Is it worth all that hacking? To be honest, I don’t mind having my IDE generate test method boilerplate so my answer would be “No”. But it was a fun experiment. :)

Lifecycle

The current implementation of dynamic tests is deliberately “raw”. One of the ways this shows is that they are not integrated into the lifecycle. From the user guide:

This means that @BeforeEach and @AfterEach methods and their corresponding extension callbacks are not executed for dynamic tests. In other words, if you access fields from the test instance within a lambda expression for a dynamic test, those fields will not be reset by callback methods or extensions between the execution of dynamic tests generated by the same @TestFactory method.

There is already an issue to address this, though.

Reflection

So what have we seen? Up to now JUnit only knew about tests that were declared at compile time. JUnit 5 has a concept of dynamic tests, which are created at run time and consist of a name and an executable that holds the test code. With that we have seen how we can create parameterized tests and use lambdas to define tests in a more modern style.

What do you think? Eager to try it out?

Share & Follow

You liked this post? Then share it with your friends and followers!
twittergoogle_plusredditlinkedin
And if you like what I'm writing about, why don't you follow me?
twittergoogle_plusrss

Other Posts