Is there any way to use pythonappend with SWIG's new builtin feature?

I have a little project that works beautifully with SWIG. In particular, some of my functions return std::vectors, which get translated to tuples in Python. Now, I do a lot of numerics, so I just have SWIG convert these to numpy arrays after they're returned from the c++ code. To do this, I use something like the following in SWIG.

%feature("pythonappend") My::Cool::Namespace::Data() const %{ if isinstance(val, tuple) : val = numpy.array(val) %}

(Actually, there are several functions named Data, some of which return floats, which is why I check that val is actually a tuple.) This works just beautifully.

But, I'd also like to use the -builtin flag that's now available. Calls to these Data functions are rare and mostly interactive, so their slowness is not a problem, but there are other slow loops that speed up significantly with the builtin option.

The problem is that when I use that flag, the pythonappend feature is silently ignored. Now, Data just returns a tuple again. Is there any way I could still return numpy arrays? I tried using typemaps, but it turned into a giant mess.

Edit:

Borealid has answered the question very nicely. Just for completeness, I include a couple related but subtly different typemaps that I need because I return by const reference and I use vectors of vectors (don't start!). These are different enough that I wouldn't want anyone else stumbling around trying to figure out the minor differences.

%typemap(out) std::vector<int>& {
  npy_intp result_size = $1->size();
  npy_intp dims[1] = { result_size };
  PyArrayObject* npy_arr = (PyArrayObject*)PyArray_SimpleNew(1, dims, NPY_INT);
  int* dat = (int*) PyArray_DATA(npy_arr);
  for (size_t i = 0; i < result_size; ++i) { dat[i] = (*$1)[i]; }
  $result = PyArray_Return(npy_arr);
}
%typemap(out) std::vector<std::vector<int> >& {
  npy_intp result_size = $1->size();
  npy_intp result_size2 = (result_size>0 ? (*$1)[0].size() : 0);
  npy_intp dims[2] = { result_size, result_size2 };
  PyArrayObject* npy_arr = (PyArrayObject*)PyArray_SimpleNew(2, dims, NPY_INT);
  int* dat = (int*) PyArray_DATA(npy_arr);
  for (size_t i = 0; i < result_size; ++i) { for (size_t j = 0; j < result_size2; ++j) { dat[i*result_size2+j] = (*$1)[i][j]; } }
  $result = PyArray_Return(npy_arr);
}

Edit 2:

Though not quite what I was looking for, similar problems may also be solved using @MONK's approach (explained here).

Access C++ shared library from Java: JNI, JNA, CNI, or SWIG? [closed]

Which of the following (or other) method would you recommend for accessing a C++ shared library from Java and why?

1. JNI: I hear this has a number of pitfalls and is quite the undertaking?
2. SWIG: Apparently this makes using JNI easier, but I've heard it has some problems too?
3. JNA: I could write a C interface and then use JNA with it instead, which is apparently significantly easier than JNI?
4. CNI: Apparently this is easier than JNI?
5. Another library: it can't be commercial and I'd prefer something that is still well maintained (this would throw out JNIEasy, JNative, and JACE - I think).

I'm trying to go for minimal development time and problems down the road. Any info on pros/cons of each of these would be appreciated.

EDIT: Optimally, I'd be able to use my C++ objects in Java, but as I mentioned in #3, I could write a C interface using just functions if absolutely necessary. The C++ interface I'd like to have available in Java is something like:

class MyObj1
{
    MyObj1(std::string, bool);
    bool Func1();
    void Func2(bool);
}

class MyObj2
{
    MyObj2(MyObj1*);
    ssize_t Func(uint8_t *, size_t, MyObj1);
}

The actual interface is a bit more complicated, but this demonstrates the type of objects and methods I'll be using. Some of the objects are derived, but the Java interface does not need to have any polymorphic behavior.

Extending python - to swig, not to swig or Cython

I found the bottleneck in my python code, played around with psycho etc. Then decided to write a c/c++ extension for performance.

With the help of swig you almost don't need to care about arguments etc. Everything works fine.

Now my question: swig creates a quite large py-file which does a lot of 'checkings' and 'PySwigObject' before calling the actual .pyd or .so code.

Does anyone of you have any experience whether there is some more performance to gain if you hand-write this file or let swig do it.

Python: SWIG vs ctypes

In python, under what circumstances is SWIG a better choice than ctypes for calling entry points in shared libraries? Let's assume you don't already have the SWIG interface file(s).

What are the performance metrics of the two?

Exposing a C++ API to Python

I'm currently working on a project were I had to wrap the C++ classes with Python to be able to script the program. So my specific experience also involved embedding the Python interpreter in our program.

The alternatives I tried were:

• Boost.Python

  I liked the cleaner API produced by Boost.Python, but the fact that it would have required that users install an additional dependency made us switch to SWIG.

• SWIG

  SWIG's main advantage for us was that it doesn't require end users to install it to use the final program.

What have you used to do this, and what has been your experience with it?

STL map in Perl using SWIG

This is duplicate of my question on SWIG mailing list.

I am trying to use stl containers in my SWIG bindings. Everything works perfectly except for stl map handling in Perl. On C++ side, I have

std::map<std::string, std::string> TryMap(const std::map<std::string, std::string> &map) {
  std::map<std::string, std::string> modified(map);
  modified["7"] = "!";
  return modified;
}

SWIG config look like this

%module stl

%include "std_string.i"

%include "std_map.i"
%template(StringStringMap) std::map<std::string, std::string>;

%{
  #include "stl.h"
%}

%include "stl.h"

In my Python script I can call TryMap this way

print dict(stl.TryMap({'a': '4'}))

and get beautiful output

{'a': '4', '7': '!'}

but in Perl I call

print Dumper stl::TryMap({'a' => '4'});

and get an error

TypeError in method 'TryMap', argument 1 of type 'std::map< std::string,std::string > const &' at perl.pl line 7.

I can actually do something like

my $map = stl::TryMap(stl::StringStringMap->new());
print $map->get('7');

and get '!', but this is not an option because there is a lot of legacy code using "TryMap" that expects normal Perl hash as its output.

I believe there is a way work this out because SWIG solves this particular problem nicely in Python and even in Perl if I use stl vectors and strings but not maps.

Is there any way to handle stl map with Perl in SWIG? I am using latest SWIG 2.0.7

UPDATE Maybe there is something wrong with perl5/std_map.i. It is too short =)

$ wc -l perl5/std_map.i python/std_map.i 
   74 perl5/std_map.i
  305 python/std_map.i

Looking for a convenient way to call Java from C++

It seems most documentation or helper libraries relating to JNI (Java Native Interface) are concerned with calling native code from Java. This seems to be the main use of it, even though it is capable of more.

I want to mostly work in the opposite direction: modify an existing (fairly large) portable C++ program by adding some Java libraries to it. For example, I want to make it call databases via JDBC, or message queue systems via JMS, or send emails, or call my own Java classes, etc. But with raw JNI this is pretty unpleasant and error-prone.

So I would ideally like to write C++ code that can call Java classes as easily as C++/CLI can call CLR classes. Something like:

using namespace java::util::regex; // namespaces mapped

Pattern p = Pattern.compile("[,\\s]+");

array<java::lang::String> result = 
    p.split("one,two, three   four ,  five");

for (int i=0; i < result.length(); i++)
    std::cout << result[i] << std::endl;

This way, I wouldn't have to manually do the work of getting the method ID by passing the name and the weird signature strings, and would be protected from programming errors caused by the unchecked APIs for calling methods. In fact it would look a lot like the equivalent Java.

NB. I AM STILL TALKING ABOUT USING JNI! As an underlying technology it is perfect for my needs. It is "in process" and highly efficient. I don't want to run Java in a separate process and make RPC calls to it. JNI itself is fine. I just want a pleasant interface to it.

There would have to be a code generation tool to make equivalent C++ classes, namespaces, methods, etc. to exactly match what is exposed by a set of Java classes I specify. The generated C++ classes would:

• Have member functions that accept similarly-wrapped versions of their parameters and then do the necessary JNI voodoo to make the call.
• Wrap the return values in the same way so I can chain calls in a natural way.
• Maintain a per-class static cache of method IDs to avoid looking up them every time.
• Be totally thread-safe, portable, open source.
• Automatically check for exceptions after every method call and produce a std C++ exception.
• Also work for when I'm writing native methods in the usual JNI way but I need to call on to other Java code.
• The array should work totally consistently between primitive types and classes.
• Will no doubt need something like global to wrap references in when they need to survive outside of a local reference frame - again, should work the same for all array/object references.

Does such a free, open-source, portable library/tool exist or am I dreaming?

Note: I found this existing question but the OP in that case wasn't nearly as demanding of perfection as I am being...

Update: a comment about SWIG led me to this previous question, which seems to indicate that it is mostly about the opposite direction and so wouldn't do what I want.

IMPORTANT

• This is about being able to write C++ code that manipulates Java classes and objects, not the other way round (see the title!)
• I already know that JNI exists (see the question!) But hand-written code to the JNI APIs is unnecessarily verbose, repetitious, error-prone, not type-checked at compile time, etc. If you want to cache method IDs and class objects it's even more verbose. I want to automatically generate C++ wrapper classes that take care of all that for me.

Update: I've started working on my own solution:

https://github.com/danielearwicker/cppjvm

If this already exists, please let me know!

NB. If you're considering using this in your own project, feel free, but bear in mind that right now the code is a few hours old, and I only wrote three very unstrenuous tests so far.

Dynamically rethrowing self-defined C++ exceptions as Python exceptions using SWIG

Situation

I want to create a Python language binding for a C++ API using SWIG. Some of the API functions may throw exceptions. The C++ application has a hierarchy of self-defined exceptions, like this example:

std::exception
  -> API::Exception
    -> API::NetworkException
      -> API::TimeoutException
      -> API::UnreachableException
    -> API::InvalidAddressException

The desired behavior is as follows:

1. All exception types should have a matching Python class as wrapper. These wrapper classes should be valid Python exceptions.

2. When an API call throws a C++ exception, it should be caught. The corresponding Python exception (i.e. the wrapper class of the caught C++ exception) should be thrown.

3. This should be a dynamic process: the Python exception type is decided at runtime, only based on the runtime type of the caught C++ exception. This way, there is no need to describe the complete exception hierarchy in the SWIG interface file.

Problems and questions

1. Wrapper classes are no Python exceptions.

   While SWIG creates wrapper classes for all self-defined exceptions (like for any other class), these classes are not Python exceptions. The wrapper of the base exception (API::Exception in the example) extends Object instead of BaseException, the Python class of which all exceptions in Python should be derived.

   Furthermore, it doesn't seem possible to let SWIG add a parent class manually. Note this is possible when using SWIG with Java through the usage of %typemap(javabase) (see SWIG documentation for details).

2. How can the Python C API throw a user-defined exception?

   The most common way to throw a Python exception from the Python C API is by calling PyErr_SetString [reference]. This is also shown in the demo application below.

   But this is only trivial with the standard (built-in) exceptions of Python, because references to them are stored in global variables [reference] in the Python C API.

   I know there is a method PyErr_NewException [reference] to get references to self-defined exceptions, but I didn't got this working.

3. How can the Python C API evaluate the C++ type at runtime and then find the corresponding Python wrapper class by name?

   I assume a Python class can be searched by name at runtime, through the reflection part of the Python C API. Is this the way to go? And how is it done in practice?

Demo application

To experiment with this problem, I created a tiny C++ API with a single function that calculates the factorial of a number. It has a minimal self-defined exception hierarchy, consisting of only one class TooBigException.

Note this exception acts as the base exception in the general problem and the application should work with any subclass of it. This means the solution may only use the dynamic (i.e. runtime) type of the caught exception to rethrow it in Python (see below).

The full source code of the demo application is as follows:

// File: numbers.h
namespace numbers {
int fact(int n);
}

// File: numbers.cpp
#include "TooBigException.h"
namespace numbers {
int fact(int n) {
    if (n > 10) throw TooBigException("Value too big", n);
    else if (n <= 1) return 1;
    else return n*fact(n-1);
}
}

// File: TooBigException.h
namespace numbers {
class TooBigException: public std::exception {
public:
    explicit TooBigException(const std::string & inMessage,
                             const int inValue);
    virtual ~TooBigException() throw() {}
    virtual const char* what() const throw();
    const std::string & message() const;
    const int value() const;
private:
    std::string mMessage;
    int mValue;
};
}

// File: TooBigException.cpp
#include "TooBigException.h"
namespace numbers {
TooBigException::TooBigException(const std::string & inMessage, const int inValue):
    std::exception(),
    mMessage(inMessage),
    mValue(inValue)
{
}
const char* TooBigException::what() const throw(){
    return mMessage.c_str();
}
const std::string & TooBigException::message() const {
    return mMessage;
}
const int TooBigException::value() const {
    return mValue;
}
}

To get the Python binding I use the following SWIG interface file:

// File: numbers.i
%module numbers
%include "stl.i"
%include "exception.i"

%{
#define SWIG_FILE_WITH_INIT
#include "TooBigException.h"
#include "numbers.h"
%}

%exception {
    try {
        $action
    }
    catch (const numbers::TooBigException & e) {
        // This catches any self-defined exception in the exception hierarchy,
        // because they all derive from this base class. 
        <TODO>
    }
    catch (const std::exception & e)
    {
        SWIG_exception(SWIG_RuntimeError, (std::string("C++ std::exception: ") + e.what()).c_str());
    }
    catch (...)
    {
        SWIG_exception(SWIG_UnknownError, "C++ anonymous exception");
    }
}

%include "TooBigException.h"
%include "numbers.h"

So every call to the API is wrapped by a try-catch block. First exceptions of our base type are caught and handled. Then all other exceptions are caught and rethrown using the SWIG exception library.

Note any subclass of numbers::TooBigException is caught and the wrappers of their dynamic (i.e. runtime) type should be thrown, not the wrapper of their static (i.e. compile time) type, which is always TooBigException!

The project can be built easily by executing the following commands on a Linux machine:

$ swig -c++ -python numbers.i
$ g++ -fPIC -shared TooBigException.cpp numbers.cpp numbers_wrap.cxx \
    -I/usr/include/python2.7 -o _numbers.so

Current implementation

My current implementation still (successfully) throws a fixed standard Python exception. The code <TODO> above is then replaced by:

PyErr_SetString(PyExc_Exception, (std::string("C++ self-defined exception ") + e.what()).c_str());
return NULL;

Which gives the following (expected) behavior in Python:

>>> import numbers
>>> fact(11)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Exception: C++ self-defined exception Value too big

Python SVN bindings for Windows

Where can I find precompiled Python SWIG SVN bindings for Windows?

How do I propagate C++ exceptions to Python in a SWIG wrapper library?

I'm writing a SWIG wrapper around a custom C++ library which defines its own C++ exception types. The library's exception types are richer and more specific than standard exceptions. (For example, one class represents parse errors and has a collection of line numbers.) How do I propagate those exceptions back to Python while preserving the type of the exception?

Python Properties & Swig

I am attempting to create python bindings for some C++ code using swig. I seem have run into a problem trying to create python properties from some accessor functions I have for methods like the following:

class Player {
public:
  void entity(Entity* entity);
  Entity* entity() const;
};

I tried creating a property using the python property function but it seems that the wrapper classes swig generates are not compatible with it at least for setters.

How do you create properties using swig?

How can I implement a C++ class in Python, to be called by C++?

I have a class interface written in C++. I have a few classes that implement this interface also written in C++. These are called in the context of a larger C++ program, which essentially implements "main". I want to be able to write implementations of this interface in Python, and allow them to be used in the context of the larger C++ program, as if they had been just written in C++.

There's been a lot written about interfacing python and C++ but I cannot quite figure out how to do what I want. The closest I can find is here: http://www.cs.brown.edu/~jwicks/boost/libs/python/doc/tutorial/doc/html/python/exposing.html#python.class_virtual_functions, but this isn't quite right.

To be more concrete, suppose I have an existing C++ interface defined something like:

// myif.h
class myif {
   public:
     virtual float myfunc(float a);
};

What I want to be able to do is something like:

// mycl.py
... some magic python stuff ...
class MyCl(myif):
  def myfunc(a):
    return a*2

Then, back in my C++ code, I want to be able to say something like:

// mymain.cc
void main(...) {
  ... some magic c++ stuff ...
  myif c = MyCl();  // get the python class
  cout << c.myfunc(5) << endl;  // should print 10
}

I hope this is sufficiently clear ;)

Why can't I get swig wrap std::vector to Ruby class?

I have an application with an embedded Ruby interpreter, and interfaces to STL classes generated by swig. Pretty much everything worked out fine thanks to swig, except for one thing:

%module Stuff
%import "std_vector.i"
namespace std
{
  %template(Vectord) vector<double>;
}; 

%inline%{
  std::vector<double> test;
%}

When I try to use this in Ruby the type Stuff::Vectord exists, but it is not the return type of the generated singleton method test. Looking at the generated C wrapper file I can see the class Vectord and its methods getting defined, but looking at _wrap_test_get I do not see anything returning sth of class Stuff::Vectord.

What do I have to do to get test typed as Vectord?

M2Crypto doesn't install in venv, or swig doesn't define __x86_64__ which breaks compiling against OpenSSL

I'm trying to install the Python M2Crypto package into a virtualenv on an x86_64 RHEL 6.1 machine. This process invokes swig, which fails with the following error:

$ virtualenv -q --no-site-packages venv
$ pip install -E venv M2Crypto==0.20.2
Downloading/unpacking M2Crypto==0.20.2
  Downloading M2Crypto-0.20.2.tar.gz (412Kb): 412Kb  downloaded
  Running setup.py egg_info for package M2Crypto
Installing collected packages: M2Crypto
  Running setup.py install for M2Crypto
    building 'M2Crypto.__m2crypto' extension
    swigging SWIG/_m2crypto.i to SWIG/_m2crypto_wrap.c
    swig -python -I/usr/include/python2.6 -I/usr/include -includeall -o SWIG/_m2crypto_wrap.c SWIG/_m2crypto.i
    /usr/include/openssl/opensslconf.h:31: Error: CPP #error ""This openssl-devel package does not work your architecture?"". Use the -cpperraswarn option to continue swig processing.
    error: command 'swig' failed with exit status 1
    Complete output from command /home/lorin/venv/bin/python -c "import setuptools;__file__='/home/lorin/venv/build/M2Crypto/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-BFiNtU-record/install-record.txt --install-headers /home/lorin/venv/include/site/python2.6:

I've got OpenSSL 1.0.0 installed via RPM packages from RedHat.

The part of /usr/include/openssl/opensslconf.h that causes the error looks like this:

#if defined(__i386__)
#include "opensslconf-i386.h"
#elif defined(__ia64__)
#include "opensslconf-ia64.h"
#elif defined(__powerpc64__)
#include "opensslconf-ppc64.h"
#elif defined(__powerpc__)
#include "opensslconf-ppc.h"
#elif defined(__s390x__)
#include "opensslconf-s390x.h"
#elif defined(__s390__)
#include "opensslconf-s390.h"
#elif defined(__sparc__) && defined(__arch64__)
#include "opensslconf-sparc64.h"
#elif defined(__sparc__)
#include "opensslconf-sparc.h"
#elif defined(__x86_64__)
#include "opensslconf-x86_64.h"
#else
#error "This openssl-devel package does not work your architecture?"
#endif

gcc has the right variable defined:

$ echo | gcc -E -dM - | grep x86_64
#define __x86_64 1
#define __x86_64__ 1

But apparenty swig doesn't, since this is the line that's failing:

swig -python -I/usr/include/python2.6 -I/usr/include -includeall -o \
  SWIG/_m2crypto_wrap.c SWIG/_m2crypto.i

Is there a way to fix this by changing something in my system configuration? M2Crypto gets installed in a virtualenv as part of a larger script I don't control, so avoiding mucking around with the M2Crypto files would be a good thing.

SWIG Python bindings to native code not working with OpenCV 2.1

I have an OpenCV project mixing Python and C. After changing to OpenCV 2.1, my calls to C code are not working any more, probably because OpenCV is no more using SWIG bindings.

From Python, I was used to call a C function with the following prototype:

int fast_support_transform(CvMat * I, CvMat * N,...);

Now, I get the following error:

TypeError: in method 'fast_support_transform', argument 1 of type 'CvMat *'

The C code is from a library created by me that uses SWIG to produces the Python interface. I'm not sure, but I think OpenCV is using ctypes now and this code is unable to send a CvMat pointer to my native code.

Do you know about a fast fix to this problem? Any tips are welcome.

UPDATE: Visitors, note this question is outdated. Python support in OpenCV is very mature now. CvMat is being represented as a Numpy array by default now.