Consider moving FormFactor unit tests to a single functional test
|Status:||Resolved||Start date:||12 May 2016|
I think that tests of form factor behaviour near singularity and along symmetry axes belongs to the functional tests, and not to unit testing.
Here are some supporting arguments
- Conceptually it is not test of a unit, but rather the model behind the whole object
- Amount of such unit tests (more than 200k of tests!)
- Execution time (more than 10sec)
- The necessity to patch google-test gives a hint that there is a misuse of the library
- Aesthetic of the resulting output
- [ RUN] OK is better than just [——]
- How one can make a screenshot for a presentation showing last lines of unit test output with total amount of tests?
- Functional test machinery is able to run a custom test
- Just any additional application as in FunctionalTest/TestCore/PolDWBAMagCylinder
Additionally, I find the new directory structure for unit test confusing
One letter in directory name (0, 1, 2, P, Q, S ?)
Naming of command files _TestCore0, _TestCore2 etc.
I personally have a slight preference for a single directory.
The minor advantage is that everything is in one place; our build server reports “321 unit tests passed” and not “26 unit tests” passed. Aesthetically, again, I’m happy to see one single table with testing results of just compiled library, but not 10 small tables here and there.
But if we want to split to several directories, why not to have same “partitioning” as in Core library itself (Algorithms, FormFactors, InputOutput, etc…). At least it will be easier to find what belongs to what.
#1 Updated by wuttke about 3 years ago
- Move extensive ff tests from Unit to Functional?
- Don't split Core/Unit tests into subdirectories, or split them differently?
The nexus between these two is: Splitting unit tests allows parallel execution. Parallel execution is desirable anyway, but particularly for the new, extensive ff tests.
#5 Updated by pospelov about 3 years ago
My previous reply disappeared since you commented same record I was commenting.
Why do we actually have a functional test framework of our own?
It's now we, it's CMake/CTest. Designed for complicated things after the build: configuring, testing, memory checking, even submitting results of test to CDash.
Of course, one can use Google Unit Test for functional testing too (in all three core/gui/python domains ;).
But the main point is about Unit <--> Functional testing. These are two different things. Heavy staff is going to functional testing (make check), that's my opinion.
#7 Updated by wuttke about 3 years ago
The number 200k comes from arbitrary choices and can be changed if time is a concern.
The tests are concerned with numeric correctness of one specific method, evaluate_for_q. In my understanding this is prototypical unit testing. The more so since you say: functional testing is »for complicated things after the build«.
Even without the new ff tests, just for the few 100 unit tests we had before, the output from the googletest default listener was overly verbous. The output generated now (91d8e8f5b) is fully sufficient. In case of failure, it still gives full information on where it happened.