Feature #1365
Feature #418: === Configuration & Build ===
Replace old ./App based performance test machinery with new one
Status: | Archived | Start date: | 04 Mar 2016 | |
---|---|---|---|---|
Priority: | Normal | Due date: | ||
Assignee: | jmfisher | % Done: | 0% | |
Category: | - | |||
Target version: | Sprint 31 |
Description
In the view of coming cleanup of ./App folder we have to create new machinery to be able to measure performance changes.
Minimal goal
- reproduce functionality of current App/TestPerformance
Suggestions/ideas:
- It has to be a python script in FunctionalTest/TestPyCore
- It has to rely mostly on SimulationRegistry/SampleBuilderFactory to construct set of standard geometries
- As possible candidates for testing geometries see App/TestPerformance
- Three general cases should be tested
- Test of core "calculations": very large detector, test runs once.
i.e. how fast our form factors, functions, algorithms are
- Test of core "overheads": standard detector (100x100), test runs 100 times
i.e. how much overhead we have to form intensity data. Important for real time simulations in GUI.
- Test of overhead on python-C++ boundary crossing
Custom form factor in python
Questions:
- Do we need actually dedicated performance test?
Can't we just rely on execution time of our current functional tests?
We could introduce a longer version of "make check" where every test will be run 5 times. So instead of 20 sec it will take 100 sec so the measurements will be more reliable.
- How much geometries should we test?
Shell we test performance of every form factor and every interference function?
- How to access system information (CPU/Memory) usage in platform independent way.
Not very important, since tests will be run mostly on Linux
- How to measure performance reliably
warming up, throttling issues, CPU time or Wall time, single threaded or multi-threaded
- How to attach performance test to buildbot
Can we reliably measure the performance by running test in container?
- Should we include also test for memory leakages, memory allocations/deallocations.
For the moment, before every release we check for leakages (and unnecessary allocation/deallocations) using valgrind and MacOS/Insturment.
History
#1 Updated by jmfisher almost 5 years ago
I have added a Python script Tests/PerformanceTests/test_performance.py which reproduces the functionality of App/TestPerformance. It has no dependencies other than libBornAgain + standard Python modules. Have not tested outside of Linux environment -- possibly needs to be modified for platform independence.
#2 Updated by jmfisher almost 5 years ago
- Status changed from Sprint to Resolved
Added custom form factor test to test_performance.py. Under *nix systems, it measures CPU time as well as wall time. The script is written in such a way that it should be easy to add additional tests and features in the future.
#3 Updated by herck over 4 years ago
- Status changed from Resolved to Archived