Refactoring #1440

Feature #1290: === Core: framework ===

Feature #1572: == Python ==

cover Python examples by functional tests

Added by wuttke over 3 years ago. Updated almost 3 years ago.

Status:ArchivedStart date:30 May 2016
Priority:HighDue date:
Assignee:wuttke% Done:

80%

Category:-
Target version:Sprint32

Description

Test should check that all examples execute without failure, and that they produce about the same detector image as before. Which is pretty much what we are currently doing in our functional test machinery. So basically, examples and tests should be merged into a common code base.

History

#1 Updated by wuttke over 3 years ago

  • Parent task deleted (#1291)

#2 Updated by wuttke over 3 years ago

  • Parent task set to #1290

#3 Updated by wuttke over 3 years ago

perhaps existing dev-tools/check-examples/check_examples.py can be used

#4 Updated by wuttke about 3 years ago

  • Priority changed from Normal to High

required for #1566 (the big renaming)

#5 Updated by wuttke about 3 years ago

  • Parent task changed from #1290 to #1572

#6 Updated by wuttke about 3 years ago

  • Status changed from New to Sprint
  • Assignee set to wuttke
  • Target version set to Sprint32

Now I know how to do it. Each example, depending on its argument, shall either plot or write to file. The latter behavior can then be used for functional testing.

#7 Updated by wuttke about 3 years ago

  • % Done changed from 0 to 80
Mostly done in ef783ff2b.
Still todo:
  • implement meaningful check for YAML output
  • write meaningful YAML output
    • for SpecularSimulations and
    • for all of Examples/python/fitting
  • remove legacy tests that are covered by examples
  • convert the other legacy tests into example form

#8 Updated by wuttke about 3 years ago

In e7f0a6d, removed redundant legacy tests. The remaining ones will be left as they are, for the time being.

#9 Updated by pospelov about 3 years ago

  • Now it is too late, but I would have a transition period about 1-2 month, when old tests are still in place, while user-examples-tests are working in parallel. [nothing to do - JWu]
    • Then we could see, if they are failing in parallel, and not independently

How to add new functional test?
how to add new user example?
Where is the code which turns user examples into functional test?
File naming convention, location of reference files
How to run single user examples via functional test machinery

  • To my opinion, user examples should not contain any visible code related to testing. They are intended for web site.
    • So I'm more for old way of user example
      def run_simulation():
          """ 
          Run simulation and plot results
          """ 
          simulation = get_simulation()
          simulation.runSimulation()
          result = simulation.getIntensityData()
          plot_results(result)
      
      if __name__ == '__main__':
          run_simulation()
      
  • If some additional testing code is required, it has to be added/injected/written in the wrapper, which runs user example
  • Another issue is not directly connected with the discussed subject, but I would like to mention it here.
    • In some user examples we are mentioning number of line of code in the description accompanying user example (e.g. http://bornagainproject.org/documentation/python_examples/rectangular_grating)
    • This line of code is mentioned in the text, and also made highlighted in the uploaded file.
    • As you know, when Python example is changed, we are uploading it manually to the Drupal, updating text field manually. And then, for all examples containing code highlights, manually changing numbers related to highlighted block. This is the most tedious part of the whole thing.

#10 Updated by wuttke about 3 years ago

Persistence test machinery is now working.

Still TODO:
  • Explain new plot-or-save function in tutorial
  • Cover remaining fitting examples
  • Let yaml files overwrite default variability threshold
  • Make examples compatible with Python3 (import future?); otherwise test under Arch fails
Nice to have (longer term):
  • Mechanism to update references to line numbers

#11 Updated by wuttke about 3 years ago

  • Status changed from Sprint to Resolved

Basically resolved; remaining work is part of Manual revision.

#12 Updated by herck almost 3 years ago

  • Status changed from Resolved to Archived

Also available in: Atom PDF