并行测试函数调用的运行时统计信息(使用pytestbenchmark或其他插件)

2024-03-29 00:11:13 发布

您现在位置:Python中文网/ 问答频道 /正文

我想使用像pytest-benchmark这样的插件来显示并行调用的运行时统计信息test_valid_submission(0),…,test_valid_submission(EXPERIMENT_SIZE)

为此,我有以下代码,它试图使用pytest-benchmark(类似于Grouping Parametrized Benchmarks with pytest)实现这一点:

@pytest.mark.parametrize("counter", range(EXPERIMENT_SIZE))
def test_performance_under_load(benchmark, counter):
    benchmark(test_valid_submission, counter)

当我打电话时

pytest --workers auto --tests-per-worker auto -vv --benchmark-only --benchmark-verbose --group-by=func

我希望最后得到a benchmark summary table, 我的EXPERIMENT_SIZE并行test_valid_submission()调用的每个运行时的最小值、最大值、平均值和标准偏差。遗憾的是,没有打印基准汇总表(请参见下面的详细信息)

@hoefling评论说pytest-benchmark不支持并行运行和收集基准数据

是否有其他pytest插件(或其他解决方案)可以

  • 收集EXPERIMENT_SIZE个并行test_valid_submission(x)调用并将它们分组在一起
  • 计算组中并行调用的运行时统计信息的最小值、最大值、平均值和标准偏差
  • 使用多个组,例如一个用于test_valid_submission(x),另一个用于test_invalid_submission(x)
  • 在测试结束时打印统计数据,类似于pytest-benchmark summary table mentioned above

关于pytest基准的详细信息

通过pytest-benchmarkEXPERIMENT_SIZE=3iterations=1rounds=1,我得到以下输出(但即使在EXPERIMENT_SIZE>=5时,它也显示rounds=5,并且没有统计数据)

但是我得到了以下输出(删除了EXPERIMENT_SIZE=3和不相关的行):

 ============================== test session starts ==========================================
 platform linux -- Python 3.6.10, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /anaconda3/envs/reg/bin/python
 benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
 plugins: repeat-0.8.0, cov-2.8.1, bdd-3.2.1, clarity-0.3.0a0, benchmark-3.2.3, parallel-0.1.0
 collected 22 items
 pytest-parallel: 8 workers (processes), 3 tests per worker (threads)

 ...

 test/test_x.py::test_valid_submission SKIPPED

Computing precision for time.perf_counter ... 50.99ns.

Calibrating to target round 5.00us; will estimate when reaching 2.50us (using: time.perf_counter, precision: 50.99ns).

Computing precision for time.perf_counter ... 48.98ns.

Calibrating to target round 5.00us; will estimate when reaching 2.50us (using: time.perf_counter, precision: 48.98ns).

Computing precision for time.perf_counter ... 49.01ns.

Calibrating to target round 5.00us; will estimate when reaching
2.50us (using: time.perf_counter, precision: 49.01ns).

Measured 1 iterations: 105.72s.   Running 5 rounds x 1 iterations ...
Measured 1 iterations: 105.73s.   Running 5 rounds x 1 iterations ...
Measured 1 iterations: 117.20s.   Running 5 rounds x 1 iterations ...   
Ran for 339.53s.   Ran for 350.11s.   Ran for 461.53s.

test/test_x.py::test_performance_under_load[2] PASSED
test/test_x.py::test_performance_under_load[1] PASSED
test/test_x.py::test_performance_under_load[0] PASSED

========================== 3 passed, 19 skipped in 714.05s (0:11:54) ========================

或者使用benchmark.pedantic(test_valid_submission, args=[counter], iterations=1, rounds=1)也不会导致打印统计数据


Tags: testsubmissionforsizetimepytestcounterprecision