sormuras.github.io

Testing In The Modular World

This is a blog about how to organize, find and execute tests. This is not an introduction to the Java module system.

tl;dr fork/clone and run sample project sormuras/testing-in-the-modular-world using Maven as its build tool.

Good ol’ times

Flash-back to the early days of unit testing in Java and to the question: “Where should I put my test files?”

For example:

src/
   com/
      xyz/
         📜 SomeClass.java
         🔨 SomeClassTests.java

While adequate for small projects, many developers felt that this approach clutters the source directory, and makes it hard to package up client binaries without also including unwanted test code, or writing unnecessarily complex packaging tasks.

main/                          test/
   com/                           com/
      xyz/                           xyz/
         📜 SomeClass.java              🔨 SomeClassTests.java

This approach allows test code to access all the public and package visible members of the classes under test.

Who made it and still makes it work today? The class-path! Every class-path element points to a root of assets contributing to the resources available at runtime. A special type of resource is a Java class which in turn declares a package it belongs to. There is no enforced restriction of how many times a package may be declared on the class-path. All assets are merged logically at runtime, effectively resulting in the same situation where classes under test and test classes reside physically in the same directory. Packages are treated as white boxes: test code may access main types as if they were placed in the same package and directory. This includes types with using package private and protected modifiers.

Ever placed a test class in a different package compared to the class under test?

main/                          test/                               test/
   com/                           com/                                black/
      xyz/                           xyz/                                box/
         📜 SomeClass.java              🔨 SomeClassTests.java              🔲 BlackBoxTests.java

Welcome (b(l)ack) to “black box testing in the package world”!

Which types and members from main are accessible from such a black box test? The answer is left open for a brush-up of the reader’s access modifier visibility memory. Hint: an accessibility table is presented later in this blog.

Fast-forward to modules

Packages are now members of modules and only some packages are exported to other modules for consumption. Extrapolating the idea of separated source set roots to the Java module system could lead to a layout like:

main/                          test/                               test/
   com.xyz/                       com.xyz/                            black.box/
      com/                           com/                                black/
         abc/                           abc/                                box/
            📜 OtherClass.java             🔨 OtherClassTests.java              🔲 BlackBoxTests.java
         xyz/                           xyz/                             ☕ module-info.java
            📜 SomeClass.java              🔨 SomeClassTests.java
      ☕ module-info.java             🔥 module-info.[java|test] 🔥

You already noticed that the white box source set contains a cloak-and-dagger module-info.[java|test] file. Before diving into this topic, let’s examine the other two plain and simple module descriptors.

module com.xyz

module com.xyz {
    requires java.logging;
    requires java.sql;

    exports com.xyz;
}

Note! The package com.abc should not be part of a module named com.xyz. Why not? See Stephen’s JPMS module naming blog for details.

open module black.box

open module black.box {
    requires com.xyz;

    requires org.junit.jupiter.api;
    requires org.assertj.core;
    requires org.mockito;
}

Black Box Testing Diagram

Black box testing is the easy part.

black-box-testing

Test module black.box is main module com.xyz’s first customer. It adheres to the modular boundaries in the same way as any other module does. The only visible package is com.xyz, package com.abc is concealed. Same goes for the modules of the external testing frameworks like JUnit, AssertJ, Mockito and others. Only their published API is use-able by test classes contained in module black.box.

Take a 2-minute-break and watch Sander Mak describing modular testing in his Java Modularity: the Year After talk. He repeats the just introduced black box testing and gives a brief outlook to white box testing.

Now to the not so easy part…

Modular White Box Testing

Let’s start the white box testing section with an enhanced accessibility table that includes columns for being in a different module.

Accessibility

The public class A in package foo with one field for every access level modifier serves as a reference. Each column lists another type and shows how access levels affect visibility. An ✅ indicates that this member of A is accessible, else ❌ is shown.

A B C D E F Access Level Modifier
package foo;            
public class A { public
  public int i; public
  protected int j; protected
  int k; no modifier or package private
  private int l; private
}            

Column E and F are already covered by modular black box testing as shown above in the open module black.box section. With F just confirming that a not exported package is not accessible from another module. But we want to write unit tests like we always did before and access internal components. We want B, C and D back! Now you may either drop the entire Java module system (for testing) or pretend your tests reside in the same module as the classes under test. Just like in the early days, when split packages were the solution. Same same but different. Because split packages are not allowed in the world of the module-path.

🔥module-info.[java|test]🔥

At least three ways exist that lift the strict module boundaries for testing.

Resort to the classpath

Delete all module-info.java files, or exclude them from compilation, and your tests ignore all boundaries implied by the Java module system. Use internal implementation details of the Java runtime, 3rd-party libraries including test framework and of course, use the internal types from your main source set. The last part was the intended goal – achieved, yes, but paid a very high price.

Let’s explore two other ways that keep boundaries of the Java module system intact.

White box modular testing with module-info.java

The foundation tool javac version 9+ and maven-compiler-plugin version 3.8.0+ support compiling module-info.java residing in test source sets.

Here you use the default module description syntax to a) shadow the main configuration and b) express additional requirements needed for testing.

// same name as main module and open for deep reflection
open module com.xyz {
    requires java.logging;          // copied from main module descriptor
    requires java.sql;              // - " -
    exports com.xyz;                // - " -

    requires org.junit.jupiter.api; // additional test requirement
    requires org.assertj.core;      // - " -
    requires org.mockito;           // - " -
}

The test module is now promoted to be the entry point for test compilation. It inherits all elements from the main module and adds additional ones. You might read is as: open (test) module com.xyz extends (main) module com.xyz Now you only need to blend in the main source set into the test module in order to make your test code resolve the classes under test.

This results in a standard Java module tuned for testing. No need to learn extra command line options which are passed to the test runtime like described in the next section.

Note: Copying parts from the main module descriptor manually is brittle. The “Java 9 compatible build tool” pro solves this by auto-merging a main and test module descriptor on-the-fly.

White box modular testing with extra java command line options

The foundation tool java version 9+ provides command line options configure the Java module system “on-the-fly” at start up time. Various test launcher tools allow additional command line options to be passed to the test runtime.

Here are the additional command line options needed to achieve the same modular configuration as above:

--add-opens                                   | "open module com.xyz"
  com.xyz/com.abc=org.junit.platform.commons  |
--add-opens                                   |
  com.xyz/com.xyz=org.junit.platform.commons  |

--add-reads                                   | "requires org.junit.jupiter.api"
  com.xyz=org.junit.jupiter.api               |
--add-reads                                   | "requires org.assertj.core"
  com.xyz=org.assertj.core                    |
--add-reads                                   | "requires org.mockito"
  com.xyz=org.mockito                         |

Before running any tests, your test classes first need to be compiled. Here build tools usually resort to the class-path and ignore the main and potentially all other module descriptors. After test compilation you need to blend in the test binaries into the main module at test runtime.

and

This option is already “supported” by some IDEs, at least they don’t stumble compiling tests when a module-info.test file is present.

White Box Testing Diagram

white-box-testing

description pending…

Test Mode

Having common names for the various black and white box testing modes described above is good basis to develop more tooling support, thus I’ll introduce a TestMode enumeration. It can be used to determine the intended test mode a user wants to execute. Or if a user wants a testing framework to execute in a specific test mode, it can be passed as a parameter.

A test mode is defined by the relation of one main and one test module name.

Test Mode Table

                          main plain    main module   main module
                             ---            foo           bar
     test plain  ---          C              B             B
     test module foo          M              A             M
     test module bar          M              M             A

Test Mode Detection Algorithm Outline

Copied from TestMode.java:

  static TestMode of(String main, String test) {
    var mainAbsent = main == null || main.trim().isEmpty(); // 11: main.isBlank();
    var testAbsent = test == null || test.trim().isEmpty(); // 11: test.isBlank();
    if (mainAbsent) {
      if (testAbsent) {      // trivial case: no modules declared at all
        return CLASSIC;
      }
      return MODULAR;        // only test module is present, no patching involved
    }
    if (testAbsent) {        // only main module is present
      return MODULAR_PATCHED_TEST_RUNTIME;
    }
    if (main.equals(test)) { // same module name
      return MODULAR_PATCHED_TEST_COMPILE;
    }
    return MODULAR;          // bi-modular testing, no patching involved
  }

Summary and Sample Projects

It depends.

It depends on what you want to test. Are you writing a standalone program that consumes modules without being designed to be re-usable itself? Is it a library you want to distribute as a Java module? Is your library distributed as a multi-release JAR? Do you test how your library behaves on the class-path and module-path?

For a library, I’d suggest the following blueprint.

Maven Blueprint

Suppose you want to write and test a module named foo in a typical single project setup: main sources are in src/main/java directory, white box test sources in src/test/java. The black box integration testing projects are located under src/it and they are executed by the maven-invoker-plugin. The simplified layout of sormuras/testing-in-the-modular-world looks like:

src
├── main
│   └── java
│       ├── foo
│       │   ├── PackageFoo.java
│       │   └── PublicFoo.java
│       └── module-info.java <------------------ module foo { exports foo; }
├── test
│   └── java                                .--- open module foo {
│       ├── foo                            /       exports foo;
│       │   └── PackageFooTests.java      /        requires org.junit.jupiter.api;
│       └── module-info.[java|test] <----<       }
└── it                                    \
    └── bar                                °---- --add-reads
        └── src                                    foo=org.junit.jupiter.api
            └── test                             --add-opens
                └── java                           foo/foo=org.junit.platform.commons
                    ├── bar
                    │   └── PublicFooTests.java
                    └── module-info.java <------ open module bar {
                                                   requires foo;
                                                   requires org.junit.jupiter.api;
                                                 }
$ mvn verify
...
[INFO] Scanning for projects...
[INFO]
[INFO]------------------------------------------------------------------------
[INFO]Building testing-in-the-modular-world 1.0-SNAPSHOT
[INFO]------------------------------------------------------------------------
[INFO]---maven-compiler-plugin:3.8.0:compile(default-compile) @testing-in-the-modular-world ---
[INFO]---maven-compiler-plugin:3.8.0:testCompile(default-testCompile) @testing-in-the-modular-world ---
[INFO]---junit-platform-maven-plugin:0.0.10:launch-junit-platform(launch) @testing-in-the-modular-world ---
[INFO] Launching JUnit Platform...
[INFO] ╷
[INFO] └─ JUnit Jupiter ✔
[INFO]    └─ PackageFooTests ✔
[INFO]       ├─ accessPackageFooInModuleFoo() ✔
...

White box tests are done.

Now module foo is installed locally and the maven-invoker-plugin executes all integration tests:

...
[INFO]---maven-jar-plugin:2.4:jar(default-jar) @testing-in-the-modular-world ---
[INFO]---maven-invoker-plugin:3.1.0:install(integration-test) @testing-in-the-modular-world ---
[INFO]---maven-invoker-plugin:3.1.0:integration-test(integration-test) @testing-in-the-modular-world ---
[INFO] Building:bar/pom.xml
[INFO]           bar/pom.xml ......................................SUCCESS (5.8 s)
[INFO]
[INFO]---maven-invoker-plugin:3.1.0:verify(integration-test) @testing-in-the-modular-world ---
[INFO]-------------------------------------------------
[INFO] Build Summary:
[INFO]   Passed: 1, Failed: 0, Errors: 0, Skipped: 0
[INFO]-------------------------------------------------
[INFO]------------------------------------------------------------------------
[INFO]BUILD SUCCESS
[INFO]------------------------------------------------------------------------

Note: although I favor the MODULAR_PATCHED_TEST_COMPILE test mode with a module-info.java describing the test module for white box testing, I recommend to stick with MODULAR_PATCHED_TEST_RUNTIME for now. Most build tools don’t support two module descriptors on the path, nor do they understand module descriptors sharing a single name.

Maven + JUnit Platform Maven Plugin

The micromata/sawdust project shows all test modes in action. Browse the sources of the sub-projects to see how to configure test mode. See also the linked Job log produced by Travis CI to verify you.

Foundation tools javac and java (and jshell)

The junit5-modular-world sample project uses Java foundation tools to demonstrate testing the modular world. This project’s layout is based on proposals introduced by the Module System Quick-Start Guide.

Resources

History

This is a living document, it will be updated now-and-then.

Cheers and Happy Testing, Christian