testsuite performace kind of "random"

[staabm]

when running the test-suite of this project on my m1 mac pro, the time and resources to execute the test-suite varies a lot

5 runs of vendor/bin/phpunit right after another without any changes to the code:

Time: 00:54.456, Memory: 1.33 GB
Time: 01:01.633, Memory: 1.35 GB
Time: 00:21.541, Memory: 751.56 MB
Time: 00:17.356, Memory: 744.00 MB
Time: 00:17.291, Memory: 734.00 MB

is this a known thing and is the cause of this variance known?

php -v
PHP 8.3.8 (cli) (built: Jun  4 2024 14:53:17) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.8, Copyright (c) Zend Technologies

[Ocramius]

the time and resources to execute the test-suite varies a lot

The test suite uses a singleton reflector to avoid re-loading all sources all the time.

Potentially, small tests that do many lookups O(n) are much faster after a larger test may have loaded all remaining sources.

I would suggest picking two different sorting seeds, and do a flame-graph out of each run, to compare.