Deep dive: Minitest

Have you ever wondered what happens when you run a Minitest test suite? How does it work?

This article will demystify Minitest’s magic by going deep into its source code. After reading it, you’ll no longer consider Minitest a magical black box and will understand how Minitest discovers your tests methods and executes them.

Reinvent the wheel

Although the standard programming wisdom instructs us to avoid it, reinventing the wheel is a great way to learn the basic principles about the wheel.

So how does Minitest work?

Let’s write a simplest code possible that demonstrates how it’s used:

require "minitest/autorun"

class MathTest < Minitest::Test
  def test_two_plus_two
    assert 2 + 2 == 4
  end
end

When we run it, we get the following output:

Run options: --seed 22395

# Running:

.

Finished in 0.000802s, 1246.8827 runs/s, 1246.8827 assertions/s.

1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

Let’s try to figure out how it works by removing the require line. If we run it without the require, we’ll get the following error:

Traceback (most recent call last):
test.rb:3:in `<main>': uninitialized constant Minitest (NameError)

It looks like we’re going to need to define that class, so let’s do that:

module Minitest
  class Test; end
end

class MathTest < Minitest::Test
  def test_two_plus_two
    assert 2 + 2 == 4
  end
end

If we run it, absolutely nothing will happen, but at least it will not blow up.

The next step is actually running the test. We need to do the following:

figure out which classes have inherited our test class
call instance methods that begin with test_

In order to do 1, we need to add some sort of descendant tracking to our Test class. Let’s add it:

module Minitest
  class Test
    def self.inherited(klass)
      @descendants ||= []
      @descendants << klass
    end

    def self.descendants
      @descendants || []
    end
  end
end

class MathTest < Minitest::Test
  def test_two_plus_two
    assert 2 + 2 == 4
  end
end

Now we’re ready for 2, so let’s call the methods that start with test_. Since these methods are instance methods, it makes sense to instantiate descendant classes we have tracked:

module Minitest
  class Test
    def self.inherited(klass)
      @descendants ||= []
      @descendants << klass
    end

    def self.descendants
      @descendants || []
    end
  end
end

class MathTest < Minitest::Test
  def test_two_plus_two
    assert 2 + 2 == 4
  end
end

Minitest::Test.descendants.each do |klass|
  klass.new.tap do |instance|
    instance.methods.grep(/^test_/).each do |method_name|
      instance.public_send(method_name)
    end
  end
end

If we run this we’ll get another error:

Traceback (most recent call last):
        6: from test.rb:22:in `<main>'
        5: from test.rb:22:in `each'
        4: from test.rb:24:in `block in <main>'
        3: from test.rb:24:in `each'
        2: from test.rb:25:in `block (2 levels) in <main>'
        1: from test.rb:25:in `public_send'
test.rb:18:in `test_two_plus_two': undefined method `assert' for #<MathTest:0x00007f8eeb0f4ef8> (NoMethodError)

This is just what we’ve expected since we didn’t define the assert method yet. Let’s add it:

module Minitest
  class Test
    def self.inherited(klass)
      @descendants ||= []
      @descendants << klass
    end

    def self.descendants
      @descendants || []
    end

    def assert(condition)
      print '.' if condition
    end
  end
end

class MathTest < Minitest::Test
  def test_two_plus_two
    assert 2 + 2 == 4
  end
end

Minitest::Test.descendants.each do |klass|
  klass.new.tap do |instance|
    instance.methods.grep(/^test_/).each do |method_name|
      instance.public_send(method_name)
    end
  end
end

Run it and witness the dot that is printed out in all its glory:

So far so good, but we need to report the number of runs, assertions and failures, in addition to dots.

Let’s count the number of assertions:

module Minitest
  class Test
    def initialize
      @assertions_count = 0
    end

    attr_reader :assertions_count

    def self.inherited(klass)
      @descendants ||= []
      @descendants << klass
    end

    def self.descendants
      @descendants || []
    end

    def assert(condition)
      @assertions_count += 1
      print '.' if condition
    end
  end
end

class MathTest < Minitest::Test
  def test_two_plus_two
    assert 2 + 2 == 4
  end
end

Minitest::Test.descendants.each do |klass|
  klass.new.tap do |instance|
    instance.methods.grep(/^test_/).each do |method_name|
      instance.public_send(method_name)
    end
  end
end

Nice, we’re now counting the assertions. The only problem now is printing it. Let’s start printing a simple report:

module Minitest
  class Test
    def initialize
      @assertions_count = 0
    end

    attr_reader :assertions_count

    def self.inherited(klass)
      @descendants ||= []
      @descendants << klass
    end

    def self.descendants
      @descendants || []
    end

    def assert(condition)
      @assertions_count += 1
      print '.' if condition
    end
  end
end

class MathTest < Minitest::Test
  def test_two_plus_two
    assert 2 + 2 == 4
  end
end

Minitest::Test.descendants.map do |klass|
  klass.new.tap do |instance|
    instance.methods.grep(/^test_/).each do |method_name|
      instance.public_send(method_name)
    end
  end
end.each_with_object(runs: 0, assertions: 0) do |instance, counter|
  counter[:assertions] += instance.assertions_count
  counter[:runs] += instance.methods.grep(/^test_/).count
end.tap do |counter|
  puts "\n\n#{counter[:runs]} runs, " \
       "#{counter[:assertions]} assertions, " \
       '0 failures, 0 errors, 0 skips'
end

Run this code, and you’ll get the report printed out:

.

1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

Great!

However, this is not precisely how Minitest works. Minitest doesn’t magically add code to the bottom of our tests to print its report. Remember, we require Minitest before our test code, so we need to keep our “test library” code before our tests.

We need to print the report when everything else has finished executing, and luckily Ruby comes with the Kernel#at_exit method we can use for this:

module Minitest
  class Test
    def initialize
      @assertions_count = 0
    end

    attr_reader :assertions_count

    def self.inherited(klass)
      @descendants ||= []
      @descendants << klass
    end

    def self.descendants
      @descendants || []
    end

    def assert(condition)
      @assertions_count += 1
      print '.' if condition
    end
  end
end

def do_the_wizardry
  Minitest::Test.descendants.map do |klass|
    klass.new.tap do |instance|
      instance.methods.grep(/^test_/).each do |method_name|
        instance.public_send(method_name)
      end
    end
  end.each_with_object(runs: 0, assertions: 0) do |instance, counter|
    counter[:assertions] += instance.assertions_count
    counter[:runs] += instance.methods.grep(/^test_/).count
  end.tap do |counter|
    puts "\n\n#{counter[:runs]} runs, " \
         "#{counter[:assertions]} assertions, " \
         '0 failures, 0 errors, 0 skips'
  end
end

at_exit { do_the_wizardry }

class MathTest < Minitest::Test
  def test_two_plus_two
    assert 2 + 2 == 4
  end
end

Excellent! Our library code is now above the test code, which is also how Minitest works and does its magic.

A peek under the hood

Since we’re now total experts when it comes to test libraries, we are no longer afraid to look under the hood of Minitest^[1].

git clone https://github.com/seattlerb/minitest.git
cd minitest
git checkout 1f2b132

Our assumption is that at_exit is used somewhere, to print the report we get when running our test suite. Let’s confirm it:

grep at_exit -nR .

This will bring us to:

lib/minitest.rb#L52-L66

def self.autorun
  at_exit {
    next if $! and not ($!.kind_of? SystemExit and $!.success?)

    exit_code = nil

    at_exit {
      @@after_run.reverse_each(&:call)
      exit exit_code || false
    }

    exit_code = Minitest.run ARGV
  } unless @@installed_at_exit
  @@installed_at_exit = true
end

Which uses at_exit to instruct Minitest to run after all the other code has been executed and the program is exiting. The first line skips invoking Minitest if the exception is raised and it’s not SystemExit with zero status. ^[2]

This hook is only set up if @@installed_at_exit has not been set, ensuring the hook will only be set up once. This allows requiring minitest/autorun multiple times and not having to worry about what will happen with the at_exit hook (imagine multiple test files each requiring minitest/autorun).

This brings us to the inside of the block:

lib/minitest.rb#L52-L66

def self.autorun
  at_exit {
    next if $! and not ($!.kind_of? SystemExit and $!.success?)

    exit_code = nil

    at_exit {
      @@after_run.reverse_each(&:call)
      exit exit_code || false
    }

    exit_code = Minitest.run ARGV
  } unless @@installed_at_exit
  @@installed_at_exit = true
end

Notice the little trick with exit_code being set to nil first, before assigning it to the result of Minitest.run. This is used to ensure that the at_exit block runs regardless of the result of Minitest.run.^[3]

This second at_exit hook will be executed after Minitest finishes its execution, so this is setting up the second layer of code that’s going to be run when the program is exiting.

In its block @@after_run is being called in reverse order. Where is it coming from?

lib/minitest.rb#L10-L15

module Minitest
  VERSION = "5.11.3" # :nodoc:
  ENCS = "".respond_to? :encoding # :nodoc:

  @@installed_at_exit ||= false
  @@after_run = []

Ah, an ordinary array, but we’re calling .call on its items. What does it store?

lib/minitest.rb#L68-L76

##
# A simple hook allowing you to run a block of code after everything
# is done running. Eg:
#
#   Minitest.after_run { p $debugging_info }

def self.after_run &block
  @@after_run << block
end

Nice, so this is how Minitest keeps track of its after_run callbacks, it appends blocks passed to Minitest.after_run to an ordinary array - nothing magical.

Now that we’ve demystified other parts of the at_exit hook that Minitest uses to deploy its magic, let’s take a look at the one thing that’s left:

lib/minitest.rb#L52-L66

def self.autorun
  at_exit {
    next if $! and not ($!.kind_of? SystemExit and $!.success?)

    exit_code = nil

    at_exit {
      @@after_run.reverse_each(&:call)
      exit exit_code || false
    }

    exit_code = Minitest.run ARGV
  } unless @@installed_at_exit
  @@installed_at_exit = true
end

This is the most important line of that method since it actually runs the tests. Let’s dig into it!

Starting the engine

Searching for it, we discover these methods:

lib/minitest.rb#L103-L144

##
# This is the top-level run method. Everything starts from here. It
# tells each Runnable sub-class to run, and each of those are
# responsible for doing whatever they do.
#
# The overall structure of a run looks like this:
#
#   Minitest.autorun
#     Minitest.run(args)
#       Minitest.__run(reporter, options)
#         Runnable.runnables.each
#           runnable.run(reporter, options)
#             self.runnable_methods.each
#               self.run_one_method(self, runnable_method, reporter)
#                 Minitest.run_one_method(klass, runnable_method)
#                   klass.new(runnable_method).run

def self.run args = []
  self.load_plugins unless args.delete("--no-plugins") || ENV["MT_NO_PLUGINS"]

  options = process_args args

  reporter = CompositeReporter.new
  reporter << SummaryReporter.new(options[:io], options)
  reporter << ProgressReporter.new(options[:io], options)

  self.reporter = reporter # this makes it available to plugins
  self.init_plugins options
  self.reporter = nil # runnables shouldn't depend on the reporter, ever

  self.parallel_executor.start if parallel_executor.respond_to?(:start)
  reporter.start
  begin
    __run reporter, options
  rescue Interrupt
    warn "Interrupted. Exiting..."
  end
  self.parallel_executor.shutdown
  reporter.report

  reporter.passed?
end

lib/minitest.rb#L146-L161

##
# Internal run method. Responsible for telling all Runnable
# sub-classes to run.

def self.__run reporter, options
  suites = Runnable.runnables.
                    reject { |s| s.runnable_methods.empty? }.
                    shuffle
  parallel, serial = suites.partition { |s| s.test_order == :parallel }

  serial.map { |suite| suite.run reporter, options } +
    parallel.map { |suite| suite.run reporter, options }
end

So far so good, but we’re little confused now about this Runnable.runnables invocation. It looks like it’s an array, but where did it come from?

We track it down:

lib/minitest.rb#L373-L378

##
# Returns all subclasses of Runnable.

def self.runnables
  @@runnables
end

This doesn’t clear things up, so we keep searching and find this code:

lib/minitest.rb#L977-L982

class Runnable # re-open
  def self.inherited klass # :nodoc:
    self.runnables << klass
    super
  end
end

Ah! It uses the Class#inherited to track all the classes that have inherited Runnable. It’s a bit weird to discover that @@runnables is being appended to, but not assigned to an array yet. Here’s the missing piece:

lib/minitest.rb#L291-L295

def self.reset # :nodoc:
  @@runnables = []
end

reset

This clears things up about Runnable.runnables being an array of classes that inherit Runnable, but we still know nothing about the nature of those classes.

Doing a quick grep for < Runnable gives us a suspect:

lib/minitest.rb#L496

class Result < Runnable

Remember that we’re still stuck at this line:

lib/minitest.rb#L146-L161

##
# Internal run method. Responsible for telling all Runnable
# sub-classes to run.

def self.__run reporter, options
  suites = Runnable.runnables.
                    reject { |s| s.runnable_methods.empty? }.
                    shuffle
  parallel, serial = suites.partition { |s| s.test_order == :parallel }

  serial.map { |suite| suite.run reporter, options } +
    parallel.map { |suite| suite.run reporter, options }
end

But we now know ^{(or do we, cough, cough?)} that Runnable.runnables is [Result]. Looks like we’re rejecting classes from that array that have empty runnable_methods.

Tracking this method down leads us to the dead end:

lib/minitest.rb#L365-L371

##
# Each subclass of Runnable is responsible for overriding this
# method to return all runnable methods. See #methods_matching.

def self.runnable_methods
  raise NotImplementedError, "subclass responsibility"
end

What!?

Turns out, this nifty re-opening of Runnable class has a purpose. The Runnable.inherited is defined on line 977, while Result < Runnable happens on line 496. Looks like the order is important here. Who knew!?

# This happens first
class Runnable
  def self.reset # :nodoc:
    @@runnables = []
  end

  reset

  def self.runnables
    @@runnables
  end
end

class Result < Runnable; end

# And then we re-open the class
class Runnable # re-open
  def self.inherited klass # :nodoc:
    self.runnables << klass
    super
  end
end

# > Runnable.runnables
# => []

Our suspect is free to go. We have found another one:

lib/minitest/test.rb#L10

  class Test < Runnable

Interesting. But where do we even require that class?

lib/minitest.rb#L987

require "minitest/test"

Ah! That’s the last line of that file, so the inherited will kick in and catch this class. This suspect is now confirmed, and we can happily declare that Runnable.runnables contains [Test].

We now track down Test.runnable_methods:

lib/minitest/test.rb#L60-L77

##
# Returns all instance methods starting with "test_". Based on
# #test_order, the methods are either sorted, randomized
# (default), or run in parallel.

def self.runnable_methods
  methods = methods_matching(/^test_/)

  case self.test_order
  when :random, :parallel then
    max = methods.size
    methods.sort.sort_by { rand max }
  when :alpha, :sorted then
    methods.sort
  else
    raise "Unknown test_order: #{self.test_order.inspect}"
  end
end

This returns all the instance methods of this class, that start with test_ by using the Runnable#methods_matching:

lib/minitest.rb#L284-L289

##
# Returns all instance methods matching the pattern +re+.

def self.methods_matching re
  public_instance_methods(true).grep(re).map(&:to_s)
end

We can now finally move a couple of lines:

lib/minitest.rb#L146-L161

##
# Internal run method. Responsible for telling all Runnable
# sub-classes to run.

def self.__run reporter, options
  suites = Runnable.runnables.
                    reject { |s| s.runnable_methods.empty? }.
                    shuffle
  parallel, serial = suites.partition { |s| s.test_order == :parallel }

  serial.map { |suite| suite.run reporter, options } +
    parallel.map { |suite| suite.run reporter, options }
end

It looks like some partitioning is happening, but we don’t care because:

lib/minitest/test.rb#L79-L85

##
# Defines the order to run tests (:random by default). Override
# this or use a convenience method to change it for your tests.

def self.test_order
  :random
end

Which will put everything in serial (parallel will be empty array) and this leads us to:

lib/minitest.rb#L146-L161

##
# Internal run method. Responsible for telling all Runnable
# sub-classes to run.

def self.__run reporter, options
  suites = Runnable.runnables.
                    reject { |s| s.runnable_methods.empty? }.
                    shuffle
  parallel, serial = suites.partition { |s| s.test_order == :parallel }

  serial.map { |suite| suite.run reporter, options } +
    parallel.map { |suite| suite.run reporter, options }
end

Which finally calls Test.run, which Test inherits from Runnable:

lib/minitest.rb#L297-L324

##
# Responsible for running all runnable methods in a given class,
# each in its own instance. Each instance is passed to the
# reporter to record.

def self.run reporter, options = {}
  filter = options[:filter] || "/./"
  filter = Regexp.new $1 if filter =~ %r%/(.*)/%

  filtered_methods = self.runnable_methods.find_all { |m|
    filter === m || filter === "#{self}##{m}"
  }

  exclude = options[:exclude]
  exclude = Regexp.new $1 if exclude =~ %r%/(.*)/%

  filtered_methods.delete_if { |m|
    exclude === m || exclude === "#{self}##{m}"
  }

  return if filtered_methods.empty?

  with_info_handler reporter do
    filtered_methods.each do |method_name|
      run_one_method self, method_name, reporter
    end
  end
end

This filters methods by the --name and --exclude options provided from the command line. Let’s get to the juicy part:

lib/minitest.rb#L319-L323

##
# Responsible for running all runnable methods in a given class,
# each in its own instance. Each instance is passed to the
# reporter to record.

def self.run reporter, options = {}
  filter = options[:filter] || "/./"
  filter = Regexp.new $1 if filter =~ %r%/(.*)/%

  filtered_methods = self.runnable_methods.find_all { |m|
    filter === m || filter === "#{self}##{m}"
  }

  exclude = options[:exclude]
  exclude = Regexp.new $1 if exclude =~ %r%/(.*)/%

  filtered_methods.delete_if { |m|
    exclude === m || exclude === "#{self}##{m}"
  }

  return if filtered_methods.empty?

  with_info_handler reporter do
    filtered_methods.each do |method_name|
      run_one_method self, method_name, reporter
    end
  end
end

Which brings us to Runnable.run_one_method:

lib/minitest.rb#L326-L335

##
# Runs a single method and has the reporter record the result.
# This was considered internal API but is factored out of run so
# that subclasses can specialize the running of an individual
# test. See Minitest::ParallelTest::ClassMethods for an example.

def self.run_one_method klass, method_name, reporter
  reporter.prerecord klass, method_name
  reporter.record Minitest.run_one_method(klass, method_name)
end

Which passes the potato to Minitest.run_one_method:

lib/minitest.rb#L959-L963

def self.run_one_method klass, method_name # :nodoc:
  result = klass.new(method_name).run
  raise "#{klass}#run _must_ return a Result" unless Result === result
  result
end

Remember that klass here is Minitest::Test and method_name is each method defined in that class that starts with test_ and has survived filtering.

Let’s repeat this line since it’s perhaps the most elegant line in the entire project.

lib/minitest.rb#L960

def self.run_one_method klass, method_name # :nodoc:
  result = klass.new(method_name).run
  raise "#{klass}#run _must_ return a Result" unless Result === result
  result
end

This line creates a new instance of Minitest::Test class and passes the current method name as the first argument. Minitest::Test inherits initialize from Minitest::Runnable which looks like this:

lib/minitest.rb#L400-L404

def initialize name # :nodoc:
  self.name       = name
  self.failures   = []
  self.assertions = 0
end

So every test method we’ve created in our original test file gets its own Minitest::Test instance, which stores the name, failures, and number of assertions. A much better approach, compared to our hack from the beginning, but there are some similarities.

The final layer

We still have one layer before we reach the end since we’re calling run on an instance of Minitest::Test.

Here is that final layer in its entirety:

lib/minitest/test.rb#L89-L110

##
# Runs a single test with setup/teardown hooks.

def run
  with_info_handler do
    time_it do
      capture_exceptions do
        before_setup; setup; after_setup

        self.send self.name
      end

      TEARDOWN_METHODS.each do |hook|
        capture_exceptions do
          self.send hook
        end
      end
    end
  end

  Result.from self # per contract
end

Let’s highlight the most important line in that method.

lib/minitest/test.rb#L98

##
# Runs a single test with setup/teardown hooks.

def run
  with_info_handler do
    time_it do
      capture_exceptions do
        before_setup; setup; after_setup

        self.send self.name
      end

      TEARDOWN_METHODS.each do |hook|
        capture_exceptions do
          self.send hook
        end
      end
    end
  end

  Result.from self # per contract
end

This calls our original test method named test_two_plus_two. The name was stored in Minitest::Runnable#initialize, as explained earlier.

Other parts of this method are pretty self-explanatory, and I will not go into details. The goal of this article is achieved since we have tracked down and found out exactly what happens when we run our Minitest tests.

Hooray!

Notes and examples

The commit 1f2b132 is no longer the latest commit in the repository. The first draft of this article was written 8 weeks ago, and I apologize for not publishing it sooner.
This means it will be skipped for any exceptions except exit 0 which raises SystemExit with success? set to true. You can test this with the following code:
```
 require 'minitest/autorun'
 exit 0
```
Which will result in the standard Minitest report being printed out. Try replacing the exit status like this:
```
 require 'minitest/autorun'
 exit 1
```
And notice that the Minitest report was not printed out. The same thing happens if you raise an exception.

The simplest example to confirm this behavior is:

 exit_code = nil

 at_exit do
   puts 'inside at_exit'
   exit exit_code
 end

 exit_code = raise

Compare that with:

 exit_code = raise

 at_exit do
   puts 'inside at_exit'
   exit exit_code
 end