JUnit 5 – Extension Model

We already know quite a lot about the next version of Java’s most ubiquitous testing framework. Let’s now look at the JUnit 5 extension model, which allows libraries and frameworks to extend JUnit with their own additions.

Overview

This post is part of a series about JUnit 5:

(If you'd rather hear me talk about JUnit 5, check out the recent vJUG session or my presentation at JavaZone.)

This series is based on the pre-release version Milestone 3 and will get updated when a new milestone or the general availability release gets published.

Most of what you will read here and more can be found in the emerging JUnit 5 user guide (that link went to the Milestone 3 version - you can find the most current version here). The code samples I show here can be found on GitHub.

JUnit 4 Extension Model

Let’s first look at how JUnit 4 solved the problem. It has two, partly competing extension mechanisms: runners and rules.

Runners

Test runners manage a test’s life cycle: instantiation, calling setup and teardown methods, running the test, handling exceptions, sending notification, etc. and JUnit 4 provides an implementation that does all of that.

In 4.0 there was only one way to extend JUnit: Create a new runner and annotate your test class with @RunWith(MyRunner.class) so that JUnit uses it instead of its own implementation.

This mechanism is pretty heavyweight and inconvenient for little extensions. And it had a very severe limitation: There could always only be one runner per test class, which made it impossible to compose them. So there was no way to benefit from the features of, e.g., both the Mockito and the Spring runners at the same time.

Rules

To overcome these limitations, JUnit 4.7 introduced rules, which are annotated fields of the test class. JUnit 4 wraps test methods (and other actions) into a statement and passes it to the rules. They can then execute some code before and after executing the statement. Additionally, test methods often call methods on rule instances during execution.

An example is the temporary folder rule:

Due to the @Rule annotation, JUnit calls folder with a statement wrapping the method testUsingTempFolder. This specific rule is written so that folder creates a temporary folder, executes the test, and deletes the folder afterwards. The test itself can then create files and folders in the temporary folder.

Other rules might run the test in Swing’s Event Dispatch Thread, set up and tear down a database, or let the test time out if it ran too long.

Rules were a big improvement but are generally limited to executing some code before and after a test is run. They can not help with extension that can’t be implemented within that frame.

State Of Affairs

JUnit has two competing extension mechanisms, each with its own limitations.

So since JUnit 4.7 there were two competing extension mechanisms, each with its own limitations but also with quite an overlap. This makes clean extension difficult. Additionally, composing different extensions can be problematic and will often not do what the developer hoped it would.

JUnit 5 Extension Model

The JUnit 5 project has a couple of core principles and one of them is to “prefer extension points over features”. This translated quite literally into an integral mechanism of the new version – not the only but the most important one for extending JUnit Jupiter. (Note that this is only relevant for the Jupiter engine; other JUnit 5 engines will not share the same extension model.)

Extension Points

JUnit 5 extensions can declare interest in certain junctures of the test life cycle. When the JUnit Jupiter engine processes a test, it steps through these junctures and calls each registered extension. In rough order of appearance, these are the extension points:

  • Test Instance Post Processor
  • BeforeAll Callback
  • Container and Test Execution Condition
  • BeforeEach Callback
  • Parameter Resolution
  • Before Test Execution Callback
  • After Test Execution Callback
  • Exception Handling
  • AfterEach Callback
  • AfterAll Callback

(Don’t worry if it’s not all that clear what each of them does. We will look at some of them later.)

Each extension point corresponds to an interface and their methods take arguments that capture the context at that specific point in the test’s lifecycle. An extension can implement any number of those interfaces and will get called by the engine with the respective arguments. It can then do whatever it needs to implement its functionality.

Extension Context

Another one of the cornerstones of the extension model is the ExtensionContext, an interface with the two minor specializations ContainerExtensionContext and TestExtensionContext. It allows extensions to access information regarding the running test and also to interact with the Jupiter machinery.

Let’s have a look at its methods to see what it has to offer:

To understand getParent() we need to peek under the hood of the Jupiter engine. During execution it creates a tree of test nodes and each node will produce these contexts. As the nodes have parents (for example the node corresponding to a test class is parent to the nodes corresponding to the method it declares), they will let their extension context reference their parent’s context.

The next block makes a test’s ID, human readable name, and tags available. These can be used to filter containers and tests.

Very importantly, the context gives access to the class or method it was created for. This allows extensions to reflectively interact with it, for example to access a test method’s annotations or a test instance’s fields.

We will not discuss JUnit’s reporting facilities in depth but suffice it to say that it is a way to log messages into different sinks, like the console or XML reports, and publishReportEntry allows an extension to interact with it.

Finally there is a store, which brings us to the next topic.

Stateless

Extensions have to be stateless

There is an important detail to consider: The engine makes no guarantees when it instantiates extensions and how long it keeps instances around. This has a number of reasons:

  • It is not clear when and how extensions should be instantiated. (For each test? For each class? For each run?)
  • JUnit does not want to bother tracking extension instances.
  • If extensions were to communicate with one another, a mechanism for exchangong data would be required anyways.

Hence, extensions have to be stateless. Any state they need to maintain has to be written to and loaded from the store that the extension context makes available. A store is a namespaced, hierarchical, key-value data structure. Let’s look at each of these properties in turn.

Namespaced

To access the store via the extension context, a Namespace must be provided (there exists an overload that does not need a namespace but it delegates to the variant that does by using a default one). The context will return a store that manages entries exclusively for that namespace. This is done to prevent collisions between different extensions operating on the same node, which could lead to accidental sharing and mutation of state.

Interestingly enough, this could also be used to intentionally access another extension’s state, allowing communication and hence interaction between extensions. That could lead to some interesting cross-library features…

Hierarchical

A store is created for each extension context, which means there is one store per node in the test tree: Each test container or test method will have its own store.

In much the same way as extension contexts point to their parents, stores point to theirs. To be more precise, when a node creates a store it hands over a reference to its parent’s store. Thus, for example, the store belonging to a test method holds a reference to the store belonging to the test class that contains the method. Upon queries (not edits!) a store will first check itself before delegating to its parent store. This makes a store’s state readable to all child stores.

Key-Value

The store itself is a simplified map, where keys and values can be of any type. Here are its most essential methods:

The methods get and remove take a type token to prevent clients from littering their code with casts. There is no magic there, the store simply does the casts internally. Overloads without type tokens exist as well as the getOrComputeIfAbsent shortcut.

Applying Extensions

After creating the extension all that is left to do is tell JUnit about it. This is as easy as adding @ExtendWith(MyExtension.class) to the test class or method that needs the extension.

Actually, a slightly less verbose and more revealing option exists. But for that we first have to look at the other pillar of JUnit’s extension model.

Custom Annotations

The JUnit 5 API is driven by annotations and the engine does a little extra work when it checks for their presences: It not only looks for annotations on classes, methods and parameters but also on other annotations. And it treats everything it finds as if it were immediately present on the examined element. Annotating annotations is possible with so-called meta-annotations and the cool thing is, all JUnit annotations are totally meta.

This makes it possible to easily create and compose annotations that are fully functional within JUnit 5:

We can then use it like this:

Or we can create more succinct annotations for our extensions:

Now we can use @Database instead of @ExtendWith(ExternalDatabaseExtension.class). And since we added ElementType.ANNOTATION_TYPE to the list of allowed targets, it is also a meta-annotation and we or others can compose it further.

An Example

Let’s say we want to benchmark how long certain tests run. First, we create the annotation we want to use:

It already points to BenchmarkExtension, which we will implement next. This is our plan:

  • to measure the runtime of the whole test class, store the time before any test is executed
  • to measure the runtime of individual test methods, store the time before a test’s execution
  • after a test’s execution retrieve the test’s launch time, compute, and print the resulting runtime
  • after all tests are executed retrieve the class’s launch time, compute, and print the resulting runtime
  • only do any of this if the class or method is annotated with @Benchmark

The last point might not be immediately obvious. Why would a method not annotated with @Benchmark be processed by the extension? This stems from the fact that if an extension is applied to a class, it automatically applies to all methods therein. So if our requirements state that we might want to benchmark the class but not necessarily all individual methods, we need to exclude them. We do this by checking whether they are individually annotated.

Coincidentally, the first four points directly correspond to four of the extension points: BeforeAll, BeforeTestExecution, AfterTestExecution, AfterAll. So all we have to do is implement the four corresponding interfaces. The implementations are pretty trivial, they just do what we said above:

Interesting details are shouldBeBenchmarked, which uses a method of JUnit’s internal API to effortlessly determine whether the current element is (meta-)annotated with @Benchmark, and storeNowAsLaunchTime/ loadLaunchTime, which use the store to write and read the launch times. Finally, the extension uses the report to log its result instead of simply printing it to the console.

You can find the code on GitHub.

The next posts will talk about conditional test execution and parameter injection and show examples for how to use the corresponding extension points. If you can’t wait, check out this post, which shows how to port two JUnit 4 rules (conditional disable and temporary folder) to JUnit 5.

Reflection

We have seen that JUnit 4’s runners and rules were not ideal to create clean, powerful, and composable extensions. JUnit 5 aims to overcome their limitations with the more general concept of extension points. They allow extensions to specify at what points in a test’s life cycle they want to intervene. We have also looked at how meta-annotations enable easy creation of custom annotations.

Share & Follow

You liked this post? Then share it with your friends and followers!
twittergoogle_plusredditlinkedin
And if you like what I'm writing about, why don't you follow me?
twittergoogle_plusrss

Other Posts