Python bindings using SWIG

The Python bindings are now generated using the standard tool Swig. See http://www.swig.org/doc.html for the official documentation. Swig is currently under active development, with the goal of supporting the C++11 standard. This is still a work in progress; the current state of C++11 support is explained in detail here: http://www.swig.org/Doc3.0/CPlusPlus11.html. There are many examples of basic Swig usage (multiple inheritance, director classes, docstrings, etc.) in the sc-intern repository under the directory technologies/python-bindings/swig.

Generating bindings during build process

Excecute cmake with argument -DBORNAGAIN_GENERATE_BINDINGS=ON

Remarks:
  • The libBornAgainCore and libBornAgainFit targets depend on the Swig wrapper file. By default, the wrapper file has no dependencies. By turning this switch ON, the C++ header files are added as a dependency to the wrapper file. This way, the wrapper will be automatically regenerated during the build process, if necessary.
  • Swig often produces warnings. These should be heeded! See below. Swig warnings may not necessarily lead to compile-time errors, but they can lead to unexpected and/or incorrect behaviour at runtime.
  • Optionally, run cmake with the additional argument -DBORNAGAIN_GENERATE_PYTHON_DOCS=ON to also regenerate the Python docstrings (extracted automatically from doxygen comments)

Adding classes to the Swig interface

Including and importing classes

  • If a class is defined in ClassName.h, simply add #include "ClassName.h" and %include "ClassName.h" in the appropriate sections of the relevant Swig interface file (libBornAgainCore.i or libBornAgainFit.i)
  • If ClassName is derived from a base class, then the header containing the definition of the base class should be visible to Swig (via %import or %include), before the ClassName.h %include statement. If not, Swig will generate a warning and the exported Python classes will have incorrect inheritance relationships.
  • If a class in libBornAgainFit needs to know about a class defined in libBornAgainCore, import the header with the argument (module="libBornAgainCore"). For example, some classes in libBornAgainFit depend on ICloneable, so libBornAgainFit.i contains the statement "%import(module="libBornAgainCore") "ICloneable.h". This allows Swig to generate the correct type information and inheritance relationships in the Python module.
  • By default, Swig ignores templated classes since Python does not support templates. To export template specializations, use the %template directive. E.g. in libBornAgainCore.i, there is the directive
    %template(kvector_t) Geometry::BasicVector3D<double>;
    

    which tells Swig to export Geometry::BasicVector3D<double> to a Python class called kvector_t.

Ignoring methods and extending classes

  • If a C++ class contains a method which should not be exported to Python, simply %ignore it. E.g.
    %ignore FitSuite::setOptions(const FitOptions&);
    %ignore FitSuite::getOptions();
    
  • It is possible to add methods to the Python-exported class which are not defined in the C++ class (or to override an %ignore'd method) using %extend:
    %extend SomeClass {
        int doSomething()
        {
            ($self)->someMethodDefinedinCPP();
            return 42;
        }
    
        int getProperty()
        {
            return ($self)->m_property;
        }
    };
    

    will generate a Python class SomeClass which will have methods doSomething() and getProperty() as above, even if the underlying C++ class has not defined these methods.
    When extending a class, '$self' is a Swig identifier which behaves like the 'this' pointer in C++.

STL containers

Swig has built-in support for most C++11 STL containers, without the need for any need for explicit conversion between to/from the analogous Python containers. Simply include the appropriate Swig header. E.g

%include "std_vector.i" // this is standard Swig header which automates conversion of std::vector into Python List type
double add_elements(std::vector<double> theList);

will generate a Python function add_elements which takes a list of Python floats as argument, automatically converts this list to an std::vector<double>, and then calls the C++ function add_elements. Swig also supports automatic conversion between std::string and Python string, via std_string.i.

Smart pointers

Swig has built-in support for boost::shared_ptr and std::shared_ptr, via boost_shared_ptr.i and std_shared_ptr.i, respectively. To declare that a class should be a smart pointer, use the %shared_ptr macro:

%include "std_shared_ptr.i" 
%shared_ptr(ISampleBuilder)

Warning

As of Swig 3.0.x, there seems to be a bug (or at least incorrect behaviour), so that under certain circumstances, if ClassName has been declared as a smart pointer, then Swig does not generate the correct bindings for functions/methods which take ClassName* as argument. This can be overcome manually using %extend:

// C++ code:
class ClassName {
  // class definitions...
public:
    void doSomething(ClassName* pOther);
};

// Swig interface
%include "std_shared_ptr.i" 
%shared_ptr(ClassName)

%include "ClassName.h" 

// manually overload ClassName::doSomething so that it takes std::shared_ptr<ClassName> argument
%extend ClassName {
    void doSomething(std::shared_ptr<ClassName> pOther)
    {
        ($self)->doSomething(pOther.get());
    }
};

Swig directors

The default behaviour of Swig is that Python classes cannot overload virtual methods inherited from C++ classes. To enable this, we have to declare classes to be "directors". E.g., to declare that all exported classes should be directors:

%feature("director");

However, this is not really needed for full functionality and generates unnecessary overhead. Instead, we can declare to be directors only those classes which are needed:
%feature("director") ICloneable;
%feature("director") IFormFactor;
// ... 

Warning

If we declare director classes on a per-class basis, we have to take inheritance relationships into account. E.g. if Class1 is director, and Class1 is derived from BaseClassA and BaseClassB, then to overload all inherited virtual methods, both BaseClassA and BaseClassB should be declared as directors. Thankfully, Swig is smart enough to detect situations like this and will generate warnings automatically.

Transfer of ownership

Classes derived from ICloneable have a method transferToCPP(). On the C++ side, this method does nothing. On the Python side, this forces Python to give up ownership of the object and to assume that the pointer will eventually be deleted on the C++ side. This is implemented using the Swig built-in method disown() (see Swig documentation for details). This is done by editing the following code in libBornAgainCore.py

class ICloneable:
    # other definitions...
    def transferToCPP(self):
        return _libBornAgainCore.ICloneable_transferToCPP(self)

to
class ICloneable:
    # other definitions...
    def transferToCPP(self):
        return self.__disown__()

This editing is done automatically during the build process by a simple search and replace.

Transfer of ownership (part II)

If newly created object is returned from C++, swig has to be instructed to take ownership on it to avoid memory leakage

in libBornAgainCore.i put the line after all #include directives but before other %include's
%newobject GISASSimulation::getIntensityData(IDetector2D::EAxesUnits units_type = IDetector2D::DEFAULT) const;

Dealing with Swig warnings

As mentioned several times above, it is very important to pay attention to any warnings from Swig during the wrapper generation. These do not always lead to compile-time error, but frequently lead to incorrect behaviour at run-time. Some common warnings include:
  • Returning pointer in director method. Often produced by classes inherited by ICloneable. This is Swig (correctly) indicating that it is dangerous to return raw pointers to Python, since it cannot ensure at compile-time that object ownership will be respected.
  • Method 'print' ignored, renaming to '_print'. Swig aims to be compatible with Python 2 and 3. Since print is a keyword in Python 2, it will automatically rename such methods to avoid clashing with keywords.
  • Overloaded operator ignored: Python operator overloading works differently than in C++, so Swig does not generate operator overloads which would be ambiguous/incorrect. This can be solved on a per-class basis by %ignore'ing the problematic overload and then %extend'ing the appropriate Python equivalent. E.g.
    %ignore SomeClass::operator[](unsigned int i);
    
    // Python's operator[] is overloaded via __getitem__ and __setitem__, so we must extend the class by hand
    %extend SomeClass {
        double __getitem__(unsigned int i)
        {
            return (*($self))[i]; // this invokes C++ operator[], even though it is ignored by Swig
        }
    
        void __setitem__(unsigned int i, double value)
        {
            (*($self))[i] = value;
        }
    };
    
  • Swig knows nothing about class 'SomeClass'; 'SomeClass' ignored. This means that a class whose definition depends on SomeClass was %included without first %including or %importing SomeClass.h. This doesn't necessarily cause compile-time errors, but can lead to incorrect runtime behaviour (e.g. if the class is derived from SomeClass)

Python 3 support

Swig generates wrapper code which should be compatible with both Python 2.7 and Python 3. Note that there is no need to regenerate the wrapper source code to accommodate Python 2/3 -- just point to the correct headers and libraries at compile time. To build against Python 3 libraries and headers, run cmake with argument -DBORNAGAIN_USE_PYTHON3. Getting cmake to find the correct Python libraries and headers on systems with multiple Python installations is a work in progress.

Python bindings using Py++

Running codegenerator.py

To generate python bindings
  • cd dev-tools/python-bindings
  • python codegenerator.py clean
  • python codegenerator.py make
  • python codegenerator.py install

Details

  • python codegenerator.py make
    • runs two different scripts MakePyCore.py and MakePyFit.py
    • they generate in current directory wrappers for libBornAgainCore and libBornAgainFit libraries
    • during these procedure some warning can be produced. They are not dangerous.
  • python codegenerator.py install
    • runs two different scripts InstallPyCore.py and InstallPyFit.py
    • they patch generated code, create Qt project file and copy everything into BornAgain source tree

Adding new class to the python API

  • Add corresponding header into Core/PythonAPI/inc/PythonCoreList.h or Fit/PythonAPI/inc/PythonFitList.h
  • Add class name into the include_classes variable in MakePyCore.py or MakePyFit.py
  • run codegenerator
    python codegenerator.py clean
    python codegenerator.py make
    python codegenerator.py install

If some it fails over some construct

When codegenerator/gccxml is not happy about something (atomic or whatever), we simply exclude that from being visibly by gccxml, by putting

#ifndef GCCXML_SKIP_THIS
std::unique_ptr<>
#endif

or
#ifndef GCCXML_SKIP_THIS
#include "IntegrationMCMiser.h" 
#endif

It's a temporary solution, but for the moment I would say it is OK.

Links

nice example of tweaking of Py++ generated code

Prerequisites

Install pyplusplus, pygccxml, gccxml

MacOS

  • Mac-Lion use

sudo port install gccxml-devel py27-pygccxml-devel py27-pyplusplus-devel

  • Mac-Maverick use

sudo port install llvm-gcc42
(obsolete) sudo port install gccxml-devel py27-pygccxml-devel py27-pyplusplus-devel
sudo port install gccxml-devel py27-pygccxml py27-pyplusplus

Linux

gccxml or CastXML

pygccxml uses one of them, provided it is in the PATH.

gccxml

Unmaintained. Does not work with (not compile under?) gcc >= 5 (https://github.com/gccxml/gccxml/issues/11), and fails in Boost.Atomic 1.55.

git clone git://github.com/gccxml/gccxml.git

Use cmake
See http://gccxml.github.io/HTML/Index.html

CastXML

Official successor of gccxml. Fully supported and preferred by pygccxml since v1.7.1. Requires modification of codegenerator.py. Workaround: symlink gccxml -> castxml. Failure from deep inside: error: unknown argument: '-fxml=cache_core.xml'

pygccxml

git clone https://github.com/gccxml/pygccxml (this seems to be new repository with some bugfix activity going on)
sudo python setup.py install

pyplusplus

See https://bitbucket.org/ompl/pyplusplus (this seems to be continuation/bugfix of old-good Py++ which works with pygccxml mentioned in previous section)
hg clone https://bitbucket.org/ompl/pyplusplus (or git-hg clone, provided by Debian package hg-fast-export)
sudo python setup.py install