PonyTest package¶

The PonyTest package provides a unit testing framework. It is designed to be as simple as possible to use, both for the unit test writer and the user running the tests.

Each unit test is a class, with a single test function. By default tests run concurrently, up to the number of scheduler threads. Tests beyond that limit are queued and started as earlier tests complete.

Each test run is provided with a helper object. This provides logging and assertion functions. By default log messages are only shown for tests that fail.

When any assertion function fails the test is counted as a fail. However, tests can also indicate failure by raising an error in the test function.

Example program¶

To use PonyTest simply write a class for each test and a TestList type that tells the PonyTest object about the tests. Typically the TestList will be Main for the package.

The following is a complete program with 2 trivial tests.

use "pony_test"

actor Main is TestList
  new create(env: Env) =>
    PonyTest(env, this)

  new make() =>
    None

  fun tag tests(test: PonyTest) =>
    test(_TestAdd)
    test(_TestSub)

class iso _TestAdd is UnitTest
  fun name():String => "addition"

  fun apply(h: TestHelper) =>
    h.assert_eq[U32](4, 2 + 2)

class iso _TestSub is UnitTest
  fun name():String => "subtraction"

  fun apply(h: TestHelper) =>
    h.assert_eq[U32](2, 4 - 2)

The make() constructor is not needed for this example. However, it allows for easy aggregation of tests (see below) so it is recommended that all test Mains provide it.

Main.create() is called only for program invocations on the current package. Main.make() is called during aggregation. If so desired extra code can be added to either of these constructors to perform additional tasks.

Test names¶

Tests are identified by names, which are used when printing test results and on the command line to select which tests to run. These names are independent of the names of the test classes in the Pony source code.

Arbitrary strings can be used for these names, but for large projects it is strongly recommended to use a hierarchical naming scheme to make it easier to select groups of tests.

You can skip any tests whose names start with a given string by using the --exclude=[prefix] command line option.

You can run only tests whose names start with a given string by using the --only=[prefix] command line option.

Aggregation¶

Often it is desirable to run a collection of unit tests from multiple different source files. For example, if several packages within a bundle each have their own unit tests it may be useful to run all tests for the bundle together.

This can be achieved by writing an aggregate test list class, which calls the list function for each package. The following is an example that aggregates the tests from packages foo and bar.

use "pony_test"
use bar = "bar"
use foo = "foo"

actor Main is TestList
  new create(env: Env) =>
    PonyTest(env, this)

  new make() =>
    None

  fun tag tests(test: PonyTest) =>
    bar.Main.make().tests(test)
    foo.Main.make().tests(test)

Aggregate test classes may themselves be aggregated. Every test list class may contain any combination of its own tests and aggregated lists.

Long tests¶

Simple tests run within a single function. When that function exits, either returning or raising an error, the test is complete. This is not viable for tests that need to use actors.

Long tests allow for delayed completion. Any test can call long_test() on its TestHelper to indicate that it needs to keep running. When the test is finally complete it calls complete() on its TestHelper.

The complete() function takes a Bool parameter to specify whether the test was a success. If any asserts fail then the test will be considered a failure regardless of the value of this parameter. However, complete() must still be called.

Since failing tests may hang, a timeout must be specified for each long test. When the test function exits a timer is started with the specified timeout. If this timer fires before complete() is called the test is marked as a failure and the timeout is reported.

On a timeout the timed_out() function is called on the unit test object. This should perform whatever test specific tidy up is required to allow the program to exit. There is no need to call complete() if a timeout occurs, although it is not an error to do so.

Note that the timeout is only relevant when a test hangs and would otherwise prevent the test program from completing. Setting a very long timeout on tests that should not be able to hang is perfectly acceptable and will not make the test take any longer if successful.

Timeouts should not be used as the standard method of detecting if a test has failed.

Exclusion groups¶

By default tests run concurrently, up to the number of scheduler threads. This may be a problem for some tests, eg if they manipulate an external file or use a system resource. To fix this issue any number of tests may be put into an exclusion group.

No tests that are in the same exclusion group will be run concurrently.

Exclusion groups are identified by name, arbitrary strings may be used. Multiple exclusion groups may be used and tests in different groups may run concurrently. Tests that do not specify an exclusion group may be run concurrently with any other tests.

The command line option "--sequential" prevents any tests from running concurrently, regardless of exclusion groups. This is intended for debugging rather than standard use.

Labels¶

Test can have label. Labels are used to filter which tests are run, by setting command line argument --label=[some custom label]. It can be used to separate unit tests from integration tests.

By default label is empty. You can set it up by overriding label(): String method in unit test.

use "pony_test"

class iso _I8AddTest is UnitTest
  fun name(): String => "_I8AddTest"
  fun label(): String => "simple"
  fun apply(h: TestHelper) =>
    h.assert_eq[I8](1, 1)

Shuffle¶

By default tests are dispatched in registration order. The --shuffle option randomizes the order, which helps detect hidden dependencies between tests. A test that only passes because another test ran first will eventually fail under shuffled ordering.

Use --shuffle to generate a random seed, or --shuffle=SEED to use a specific U64 seed for reproducibility. When shuffle is active, the seed is printed before any test output:

Test seed: 8675309

To reproduce a failure, pass the seed back: --shuffle=8675309. The same seed always produces the same test ordering.

For CI environments that need sequential execution, --sequential --shuffle is the recommended combination: you get stable runs without resource contention, and each run uses a different seed so test coupling surfaces over time.

--list --shuffle=SEED prints the test names in the shuffled order that the given seed would produce.

Property-Based Testing¶

PonyTest includes property-based testing. A property test generates random inputs and verifies that a property holds for all of them. When a property fails, the failing input is automatically shrunk to a minimal reproducing case.

Standalone property¶

Define a class that implements Property[T] and register it:

use "pony_test"

class iso _ListReverseProperty is Property[Array[USize]]
  fun name(): String => "list/reverse"

  fun gen(): Generator[Array[USize]] =>
    Generators.seq_of[USize, Array[USize]](Generators.usize())

  fun ref property(arg1: Array[USize], h: PropertyHelper) =>
    h.assert_array_eq[USize](arg1, arg1.reverse().reverse())

Register it in your test list:

fun tag tests(test: PonyTest) =>
  test.property(_ListReverseProperty)

Inline property¶

For quick one-off checks, use for_all directly from a UnitTest:

class iso _MyTest is UnitTest
  fun name(): String => "my_test"

  fun apply(h: TestHelper) ? =>
    h.for_all[U8](recover Generators.u8() end)(
      {(u, h) => h.assert_true(u <= U8.max_value()) })?

Multi-argument properties¶

Property2[T1, T2], Property3[T1, T2, T3], and Property4[T1, T2, T3, T4] test properties with multiple generated arguments. Corresponding for_all2, for_all3, and for_all4 methods are available on TestHelper.

Stateful properties¶

StatefulProperty[S, M, Cmd] tests stateful systems by generating sequences of commands, checking an invariant after each step, and running a final check. Register with test.stateful_property().

Classification¶

Property tests can collect distribution statistics to verify that generated inputs cover the intended range. Call h.classify(label) to tag each sample, h.tabulate(heading, label) for cross-tabulation, or h.cover(condition, label, min_pct) to require that at least min_pct percent of samples satisfy a condition. Statistics print after all samples run.

Regression persistence¶

When a property test fails, the failing input's choice sequence is saved to .ponytest/<property-name>.choices. On the next run, the saved regression replays first: if it still fails, the test fails immediately; if it passes, the saved regression is cleared.

Set PONYTEST_NO_DB=1 to disable persistence entirely, or PONYTEST_DB_DIR=path to change the storage directory.

Setting up and tearing down a test environment¶

Set Up¶

Any kind of fixture or environment necessary for executing a UnitTest can be set up either in the tests constructor or in a function called set_up().

set_up() is called before the test is executed. It is partial, if it errors, the test is not executed but reported as failing during set up. The test's TestHelper is handed to set_up() in order to log messages or access the tests Env via TestHelper.env.

Tear Down¶

Each unit test object may define a tear_down() function. This is called after the test has finished to allow tearing down of any complex environment that had to be set up for the test.

The tear_down() function is called for each test regardless of whether it passed or failed. If a test times out tear_down() will be called after timed_out() returns.

When a test is in an exclusion group, the tear_down() call is considered part of the tests run. The next test in the exclusion group will not start until after tear_down() returns on the current test.

The test's TestHelper is handed to tear_down() and it is permitted to log messages and call assert functions during tear down.

Example¶

The following example creates a temporary directory in the set_up() function and removes it in the tear_down() function, thus simplifying the test function itself:

use "pony_test"
use "files"

class iso TempDirTest
  var tmp_dir: (FilePath | None) = None

  fun name(): String => "temp-dir"

  fun ref set_up(h: TestHelper)? =>
    tmp_dir = FilePath.mkdtemp(FileAuth(h.env.root), "temp-dir")?

  fun ref tear_down(h: TestHelper) =>
    try
      (tmp_dir as FilePath).remove()
    end

  fun apply(h: TestHelper)? =>
    let dir = tmp_dir as FilePath
    // do something inside the temporary directory

Custom reporters¶

By default, PonyTest prints results to stdout. To receive structured results programmatically, pass a TestReporter to the PonyTest constructor.

A TestReporter is any actor with three behaviours:

test_started(name: String) — a test began executing.
test_complete(result: TestResult val) — a test finished. TestResult carries the test name, pass/fail status, and log messages. Results arrive in completion order.
testing_complete(results: Array[TestResult val] val) — all tests finished. The array is in registration order.

use "pony_test"

actor MyReporter
  be test_started(name: String) =>
    // ...

  be test_complete(result: TestResult val) =>
    // ...

  be testing_complete(results: Array[TestResult val] val) =>
    // ...

actor Main is TestList
  new create(env: Env) =>
    PonyTest(env, this, MyReporter)

  new make() => None

  fun tag tests(test: PonyTest) =>
    test(_MyTest)

When a reporter is supplied, PonyTest does not set the process exit code. The reporter receives the results and the caller decides how to handle failures. Without a reporter, PonyTest calls env.exitcode(-1) when any test fails.

Two built-in reporters simplify meta-testing (running a sub-PonyTest from within a test):

PassReporter(h) completes the outer TestHelper when all sub-tests pass.
FailReporter(h) completes the outer TestHelper when any sub-test fails.