Feature #1365

Feature #418: === Configuration & Build ===

Replace old ./App based performance test machinery with new one

Added by pospelov over 4 years ago. Updated about 4 years ago.

Status:ArchivedStart date:04 Mar 2016
Priority:NormalDue date:
Assignee:jmfisher% Done:

0%

Category:-
Target version:Sprint 31

Description

In the view of coming cleanup of ./App folder we have to create new machinery to be able to measure performance changes.

Minimal goal

  • reproduce functionality of current App/TestPerformance

Suggestions/ideas:

  • It has to be a python script in FunctionalTest/TestPyCore
  • It has to rely mostly on SimulationRegistry/SampleBuilderFactory to construct set of standard geometries
  • As possible candidates for testing geometries see App/TestPerformance
  • Three general cases should be tested
    • Test of core "calculations": very large detector, test runs once.

i.e. how fast our form factors, functions, algorithms are

  • Test of core "overheads": standard detector (100x100), test runs 100 times

i.e. how much overhead we have to form intensity data. Important for real time simulations in GUI.

  • Test of overhead on python-C++ boundary crossing

Custom form factor in python

Questions:

  • Do we need actually dedicated performance test?

Can't we just rely on execution time of our current functional tests?

We could introduce a longer version of "make check" where every test will be run 5 times. So instead of 20 sec it will take 100 sec so the measurements will be more reliable.

  • How much geometries should we test?

Shell we test performance of every form factor and every interference function?

  • How to access system information (CPU/Memory) usage in platform independent way.

Not very important, since tests will be run mostly on Linux

  • How to measure performance reliably

warming up, throttling issues, CPU time or Wall time, single threaded or multi-threaded

  • How to attach performance test to buildbot

Can we reliably measure the performance by running test in container?

  • Should we include also test for memory leakages, memory allocations/deallocations.

For the moment, before every release we check for leakages (and unnecessary allocation/deallocations) using valgrind and MacOS/Insturment.

History

#1 Updated by jmfisher over 4 years ago

I have added a Python script Tests/PerformanceTests/test_performance.py which reproduces the functionality of App/TestPerformance. It has no dependencies other than libBornAgain + standard Python modules. Have not tested outside of Linux environment -- possibly needs to be modified for platform independence.

#2 Updated by jmfisher over 4 years ago

  • Status changed from Sprint to Resolved

Added custom form factor test to test_performance.py. Under *nix systems, it measures CPU time as well as wall time. The script is written in such a way that it should be easy to add additional tests and features in the future.

#3 Updated by herck about 4 years ago

  • Status changed from Resolved to Archived

Also available in: Atom PDF