Test suite should be run in random order #3610

New issue

Open

opened 2026-02-20 16:07:40 -05:00 by deekerman · 3 comments

deekerman commented

2026-02-20 16:07:40 -05:00

Owner

Originally created by @connorshea on GitHub (Mar 4, 2019).

Issue

Essentially, this: https://qualitycoding.org/random-test-order/

Expected Behavior

The test suite should be run with a randomly generated seed to guarantee that tests don't rely on each other's state (essentially, if you assume A -> B -> C -> D, and C relies on something B adds to the database – unintentionally or otherwise – C will now fail when run in isolation but not when run as part of the test suite.)

Similarly, it can cause problems when introducing new tests that mess with the existent order and reveal that tests have been unintentionally reliant on state from other tests.

It also could potentially mean that a test doesn't actually test the functionality, it's just been reliant on state that was introduced by a previous test (e.g. A creates a User and then doesn't get rid of it. B runs, and the test tries to create a User in a different way that doesn't work, but the database still has the User that A created, so B checks if there's a User in the db and passes despite the code being broken).

Actual Behavior

The tests are run without a seed, so they always run in the exact same order every time.

Possible Fix

The best fix would probably be to use Codeception's shuffle feature by setting shuffle: true in the codeception.yml file.

You may want to run the test suite a few times with shuffle: true to see if that brings up any existing state problems that would result in random test failures. Ultimately, I'd say this'd be a significant improvement over the current situation even if it may introduce flaky tests occasionally, since it's exposing an anti-pattern.

Steps to Reproduce

Look at Travis CI logs and note that the tests are always run in the same order, and that the test runner reports the seed as being blank.

Originally created by @connorshea on GitHub (Mar 4, 2019). #### Issue Essentially, this: https://qualitycoding.org/random-test-order/ #### Expected Behavior The test suite should be run with a randomly generated seed to guarantee that tests don't rely on each other's state (essentially, if you assume A -> B -> C -> D, and C relies on something B adds to the database – unintentionally or otherwise – C will now fail when run in isolation but not when run as part of the test suite.) Similarly, it can cause problems when introducing new tests that mess with the existent order and reveal that tests have been unintentionally reliant on state from other tests. It also could potentially mean that a test doesn't _actually_ test the functionality, it's just been reliant on state that was introduced by a previous test (e.g. A creates a User and then doesn't get rid of it. B runs, and the test tries to create a User in a different way that doesn't work, but the database still has the User that A created, so B checks if there's a User in the db and passes despite the code being broken). #### Actual Behavior The tests are run without a seed, so they always run in the exact same order every time. #### Possible Fix The best fix would probably be to use [Codeception's shuffle feature](https://codeception.com/docs/reference/Configuration) by setting `shuffle: true` in the `codeception.yml` file. You may want to run the test suite a few times with `shuffle: true` to see if that brings up any existing state problems that would result in random test failures. Ultimately, I'd say this'd be a significant improvement over the current situation even if it may introduce flaky tests occasionally, since it's exposing an anti-pattern. #### Steps to Reproduce Look at Travis CI logs and note that the tests are always run in the same order, and that the test runner reports the seed as being blank.

deekerman added the

Status:Fix Proposed

Type:Suggestion

labels

2026-02-20 16:07:40 -05:00

deekerman commented

2026-02-20 16:07:41 -05:00

Author

Owner

@Dillon-Brown commented on GitHub (Mar 6, 2019):

Very much agreed, I think this will take some time before we can implement it successfully though. Our Travis builds are failing pretty randomly as it is.

@Dillon-Brown commented on GitHub (Mar 6, 2019): Very much agreed, I think this will take some time before we can implement it successfully though. Our Travis builds are failing pretty randomly as it is.

deekerman commented

2026-02-20 16:07:41 -05:00

Author

Owner

@connorshea commented on GitHub (Mar 6, 2019):

Have you looked at the docs on risky tests? It seems like it could be useful to enable some of these to make sure the tests aren't messing with global scope and such: https://phpunit.de/manual/6.5/en/risky-tests.html

@connorshea commented on GitHub (Mar 6, 2019): Have you looked at the docs on risky tests? It seems like it could be useful to enable some of these to make sure the tests aren't messing with global scope and such: https://phpunit.de/manual/6.5/en/risky-tests.html

deekerman commented

2026-02-20 16:07:41 -05:00

Author

Owner

@lazka commented on GitHub (Mar 27, 2019):

I'v tried to improve the phpunit tests for this in #7113.

Not motivated by random order, but because many tests failed if run separately and not as part of the whole test suite. But the solution is the same.

@lazka commented on GitHub (Mar 27, 2019): I'v tried to improve the phpunit tests for this in #7113. Not motivated by random order, but because many tests failed if run separately and not as part of the whole test suite. But the solution is the same.