Refactoring #248

PythonAPI: redesing Py++ boost::python API code generation and installation

Added by pospelov over 7 years ago. Updated over 7 years ago.

Status:ArchivedStart date:30 Mar 2013
Priority:NormalDue date:
Assignee:pospelov% Done:

0%

Category:-
Target version:Sprint 12

Description

  • find the reason of huge memory consumption during automatic boost python code generation (pygccxml)

History

#1 Updated by pospelov over 7 years ago

  • Status changed from Backlog to Sprint

#2 Updated by pospelov over 7 years ago

  • Target version set to Sprint 12

#3 Updated by pospelov over 7 years ago

  • Assignee set to pospelov

#4 Updated by pospelov over 7 years ago

  • Tracker changed from Bug to Refactoring
  • Subject changed from PythonAPI: find the reason of huge memory consumption during autmatic boost python code generation (pygccxml) to PythonAPI: redesing Py++ boost::python API code generation and installation
  • Description updated (diff)
  • The main problem for boost/Py++ is the presence of pure virtual functions.

Ideally I would like to expose to python only methods which are necessary for sample creation, running the simulation and retrieving the results.
However, If I'm excluding all unnecessary methods from Py++ consideration, including pure virtual methods, than Py++ produces the boost::python code which is not compilable.

The problem is in a way boost implements for Python mechanism of function overloading. According to the boost we have to provide lots of additional wrapping classes (it's where we use Py++ to generates them).
PyObject is constructed from our class under consideration in a way:
boost::class_<Wrapper_of_Base<Base> > x;

Here 'Base' is our class, 'Wrapper_Of_Base' is Py++ generated wrapper which contains overloading magic, boost::class_<Wrapper_Of_BAse> is our future PyObject

If I exclude in Py++ virtual methods for Base class, than generated code of Wrapper_of_Base will not have anything about these virtual methods too.
As a result, compiler complains that we are trying instantiate template from a class without implementing pure virtual methods.

That means, that we have to expose all pure virtual methods (and, in principle, we have to expose all types in method's arguments).

virtual void getInterference(IInterference *interference) = 0;
(both, getInterference method and IInterference class should be exposed)

That's a lot.

#5 Updated by pospelov over 7 years ago

PythonAPI generation has been fully redesigned.

Now it resides in dev-tools/python-bindings

See instruction how to use here

List of significant changes
  • now regeneration of the whole PythonAPI takes 20 sec of CPU and 200 Mb of memory instead of 5 min and 4Gb of memory
  • code generation works under Linux too
  • installation of generated code into BornAgain source tree is more gentle. There is no replacement of unchanged files so recompilation is faster.
  • as a side ane very pleasant effect - shared_ptr are working from python
    • so no technical reasons to get rid from PTransform3D and shared_ptr to make PythonAPI working
    • with remainder, that we have to set global policy about shared_ptr in BornAgain in the view of multi threading...
    • ...and make lots of tests before actually starting to modify interfaces
  • All Python, C++ functional tests are working

#6 Updated by pospelov over 7 years ago

  • Status changed from Sprint to Resolved

#7 Updated by herck over 7 years ago

  • Status changed from Resolved to Archived

Also available in: Atom PDF