Nicolas Grekas
Contributed by Nicolas Grekas in #16194

Transient tests are those which fail randomly depending on spurious and external circumstances, such as the underlying system load. These tests are very risky because they make your test suite unreliable.

Tests that deal with time-related functions are one of the most common transient tests. Consider for example the following test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
use Symfony\Component\Stopwatch\Stopwatch;

class MyTest extends \PHPUnit_Framework_TestCase
{
    public function testSomething()
    {
        $stopwatch = new Stopwatch();

        $stopwatch->start('event');
        sleep(10);
        $duration = $stopwatch->stop()->getDuration();

        $this->assertEquals(10, $duration);
    }
}

This code is so simple that it seems impossible to fail. However, depending on the load of the server, the $duration could be for example 10.00000023 and the test would fail for no apparent reason.

This kind of errors happen frequently when using public continuous integration services like Travis CI. We even have a long-running issue to hunt all these transient tests.

Clock mocking

In order to solve all the time-related test errors, the PHPUnit bridge now includes a ClockMock class which can be used in your PHPUnit tests. This class replaces the PHP's built-in time(), microtime(), sleep() and usleep() functions by its own implementations.

This means that you don't need to make a single change in your original code, except when using new DateTime(), which must be replaced by DateTime::createFromFormat('U', time()) to use the mocked time() function.

The clock mocking is enabled on demand for the tests which need it. The recommended way to enable it is to add a special @group annotation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
/**
 * @group time-sensitive
 */
class MyTest extends \PHPUnit_Framework_TestCase
{
    public function testSomething()
    {
        $stopwatch = new Stopwatch();

        $stopwatch->start('event');
        sleep(10);
        $duration = $stopwatch->stop()->getDuration();

        $this->assertEquals(10, $duration);
    }
}

And that's all! This test will never fail again because of getting a wrong time-related result. The sleep(10) call will make the clock advance 10 exact seconds and your test will always pass.

An added bonus of using the ClockMock class is that time passes instantly. Using PHP's sleep(10) will make your test wait for 10 actual seconds (more or less). In contrast, the ClockMock class advances the internal clock the given number of seconds without actually waiting that time, so your test will execute 10 seconds faster.

Alternative Ways to Enable the Clock Mocking

The @group time-sensitive works "by convention" and assumes that the namespace of the tested class can be obtained just by removing the \Tests\ part from the test namespace.

If this convention doesn't work for your application, you can also configure the mocked namespaces in the phpunit.xml file, as done for example in the HttpKernel component:

1
2
3
4
5
6
7
8
9
<listeners>
    <listener class="Symfony\Bridge\PhpUnit\SymfonyTestsListener">
        <arguments>
            <array>
                <element><string>Symfony\Component\HttpFoundation</string></element>
            </array>
        </arguments>
    </listener>
</listeners>

Lastly, you can also enable clock mocking explicitly. Just call the \Symfony\Bridge\PhpUnit\ClockMock::register() method from setupBeforeClass() and pass the FQCN from which the convention explained before should be applied. Then, pass a boolean argument to ClockMock::withClockMock() method to enable/disable the clock mocking.

Published in #Living on the edge