Creation of new functional StandardTest

Overview

By standard test we understand a test case, consisting of sample, instrument and simulation definitions, that is meant to be used in three functional standard test suites:
  • Core: run simulation, and check result against reference data
  • Python: export simulation to Python, run Python script, and check result against runtime C++ simulation
  • GUI: transform simulation to XML representation, transform back test into core object, run simulation, and check against result from running the original core simulation.
The main programs for these simulations are in
  • Tests/Functional/Core/CoreStandardTest.cpp
  • Tests/Functional/PyCore/export/PyStandardExportTest.cpp
  • Tests/Functional/GUI/GUIStandardTest.cpp
The reference data are in
  • Tests/ReferenceData/StandardSuite/
When a Core simulation fails, output data are written to
  • <build>/ref/StandardSuite/
Reference data files have names of the form <testcase>.<datatype>[.gz], where
  • <testcase> is the name of the standard test (e.g. MultiLayerWithRoughness)
  • <datatype> is 'int' for intensity maps, and 'yaml' for any other hierarchical data.

Creation of standard functional test

Create new SampleBuilder representing new geometry you are going to test
  • place it in Core/StandardSamples
  • register the builder in Core/StandardSamples/SampleBuilderFactory
If necessary, create new simulation description, or use already existing one
  • simulation with the registered name "MiniGISAS" is normally used for functional testing
  • see Core/StandardSamples/StandardSimulations and Core/StandardSamples/SimulationRegistry for already existing simulations

Create new functional test description in Testachinery/StandardSimulationsRegistry. Specify the following:

    add("MyNewTest",                // unique name of the test
        "My new test description",  // one-line description
        "MiniGISAS",                // simulation name
        "MyNewTestBuilder",         // sample builder name
        "None",                     // or component registry, see next section
        2e-10);                     // variability to be admitted when checking against another data set

For all three test suites, edit CMakeLists.txt to add the test name to the command "set(test_cases". File locations are
  • Tests/Functional/Core/CMakeLists.txt
  • Tests/Functional/PyCore/export/CMakeLists.txt
  • Tests/Functional/GUI/CMakeLists.txt

Rebuild BornAgain, and run "ctest [-j<n>] [-R <expr>]" where <expr> is a regular expression that restricts execution to matching tests.

The new test will fail due to the absence of the reference file. See above (overview) for the location of the test output. Copy the test output to the reference directory, and rerun ctest. If all tests pass, put the new reference file under git control. Intensity maps should be gzip'ed.

Standard functional test with subtests.

A functional test may comprise a set of subtests that differ from each other in one sample component. For example, the tests "FormFactors" and "FormFactorAbs" comprise over 20 subtests where everything stays the same, except for the particle form factors that vary from subtest to subtest.

The sequential execution of the subtests is implemented in IStandardTest::execute_subtests(). See there how the subtest registry works.

To create standard functional test with subtests, do the following:
  • Create corresponding SampleBuilder
    • Use Core/StandardSamples/ParticleInTheAirBuilder as an example. This builder uses external component service to get form factor of the particle.
  • Create corresponing component registry which will provide your tests with necessary components
    • Add code to TestMachinery/SubtestRegistry.h|cpp.
  • Register the test as explained in the previous section, specifying the component registry in place of "None":
        add("MyNewTest",
            "My new test description",
            "MiniGISAS",
            "MyNewTestBuilder",
            "MyComponentRegistry", // the name of the component registry as hard-coded in SubtestRegistry
            2e-10);