Our test suite was fully green. Thousands of tests, zero failures, after merging a long-diverged feature branch into a large multi-module project. Then we ran it backwards.

The check

Passing once, in one order, is not the same as passing. Maven Surefire has a runOrder property that controls the sequence in which test classes execute, and it defaults to filesystem order (roughly alphabetical, depending on your OS). As an extra confidence check after the merge, before trusting the green build, we reran the same suite with the order reversed:

mvn test -Dsurefire.runOrder=reversealphabetical

reversealphabetical is a real value; the valid options are alphabetical, reversealphabetical, random, hourly, failedfirst, balanced, filesystem, and testorder. Note that reversed is not one of them: it’s an easy value to guess and get wrong.

Nothing about this requires new tooling: no dependency, no test-framework change, no CI pipeline redesign, just a JVM property flip on a command you already run.

What broke

Two test classes in the same module, which passed cleanly under the default order, failed under the reversed one. The three symptoms looked unrelated at first:

  • A test expecting specific integer values from a lookup/config layer got null instead.

  • A test expecting a specific exception to be thrown didn’t get one.

  • A test expecting an Integer-typed value got a BigDecimal instead, the wrong type inferred from stale schema/column metadata.

Three different assertions, three different failure shapes, one root cause.

The shared root cause

The project has a metadata/schema cache implemented as a JVM-wide singleton, registered in a global cache registry and lazily populated on first access from whichever database connection happens to be JNDI-bound at that moment. Maven Surefire, by default, reuses forked JVMs across test classes (reuseForks=true) for performance. Each test class in this suite rebinds its own throwaway in-memory database via JNDI in @BeforeClass, but nothing ever told the singleton cache to forget what it had already loaded from a previous class’s database.

Under the default order, whichever class happened to run first primed the cache with data that, by coincidence, matched what the later classes needed. Under the reversed order, a different class primed the cache first, with different or incompatible data. The affected tests didn’t fail loudly on a missing bind; they silently read stale, wrong metadata left over from a class that had already torn down its database.

This is the failure mode to watch for whenever JVM reuse meets shared mutable state: a "passing" test isn’t proving its own logic is correct, it’s proving that whatever ran before it in the same JVM happened to leave behind compatible state.

The fix

Two lines, added to the affected test classes' @BeforeClass, immediately after rebinding the JNDI datasource:

resetDataSourceLookup();
cacheRegistry.clear();

Drop the cached datasource lookup and clear every cached entry in the registry, so the next access reloads fresh from this class’s database instead of serving whatever the previous class left behind.

The pattern already existed elsewhere in the codebase: one test class had it, most didn’t. The underlying hazard had been discovered before and patched ad hoc, but never turned into a rule, so it silently regressed as new test classes got copy-pasted from ones that lacked it.

After adding it, both the reversed-order and default-order runs passed clean at module scope and at full multi-module reactor scope: BUILD SUCCESS, roughly 3,900 tests, zero failures, in either order.

The reset guards against the symptom, not the cause. The actual fix is not having a JVM-wide singleton cache reachable from test setup at all, and we didn’t do that here: the cache is threaded through a large, long-diverged monolith, and auditing every caller before removing it is its own project, not something you fold into a merge. The two-line reset is the stopgap until that happens.

Why this generalizes

The combination that caused this isn’t specific to this codebase:

  • A JVM-wide singleton or static cache, populated lazily.

  • A test runner that reuses forked JVMs across test classes, which is Surefire’s default and a common default elsewhere too.

  • Test classes that assume class-level isolation without enforcing it.

Any JVM test suite with that combination is exposed to the same bug class, regardless of what the cache holds or what framework wires up the JNDI resources. The fix is equally generic: explicit state reset in @BeforeClass or @Before, a standard JUnit idiom, not a project-specific workaround.

Takeaway

"All green" under one fixed execution order is not proof of test independence. It’s proof the tests happen to agree with whatever order you always run them in. -Dsurefire.runOrder=reversealphabetical costs one flag and falsifies that assumption directly. It’s cheap enough to run occasionally, and worth running at least once before trusting a big merge or refactor: the three symptoms above looked like three unrelated bugs until reversing the order forced them to share a stack trace.