Linking to ibverbs

The Python wheels for spead2 support ibverbs. But what happens if the ibverbs library isn’t installed on the user’s machine? While for pcap we simply bundle a copy of the library into the wheel, this is problematic for ibverbs: it uses configuration files in /etc/libibverbs.d to configure drivers, which are themselves contained in other shared libraries. At best, one would end up with a mix of code from the wheel and from the operating system, which could easily lead to compatibility problems.

Instead, the ibverbs libraries (libibverbs.so, librdmacm.so and libmlx5.so) are loaded dynamically at runtime with dlopen(3). If the library is not found, the corresponding functionality in spead2 is disabled.

Unfortunately, dynamic loading is not easy to use: dlsym(3) returns a raw pointer with no type information. One needs to check for errors and cast the pointer to the proper type before it can be invoked. This leads to a lot of boilerplate code, which in spead2 is generated at build time by gen/gen_loader.py for each of the libraries.

The process starts with the signatures for the functions that are to be written, which are parsed using pycparser. The actual header files for these functions are not suitable for pycparser (it only handles standard C99), so the script has embedded declarations for all the functions we wish to wrap. It then emits a header file and a source file, both generated from templates using jinja2. It generates a private initialisation function that opens the library (if possible) and extracts the symbols from it. It also generates the following for each function:

• A declaration of the function pointer. It has the same name as the original function, but is in the spead2 namespace to prevent symbol conflicts.

• An implementation that calls the initialisation function (using std::call_once(), to avoid initialising multiple times), then calls through the function pointer. The function pointer is defined to initially point to this implementation, so that the first use of the function pointer does the initialisation.

• A “stub” implementation. If there was a problem loading the library, the function pointers are all changed to point to the stub implementations, which when called will re-throw the exception that occurred during initialisation.

Additionally, some functions are optional: they were added in later versions of the wrapped library, and spead2 has fallback code that can be used if the function is not available. For those, the generator additionally creates:

• A “missing” implementation, which throws a std::system_error with error code EOPNOTSUPP. If the function is not found during initialisation, the function pointer is set to point at this implementation.

• A (public) function which has the same name but prefixed with has_, which returns a boolean to indicate whether the function is present.

Changing a function pointer during initialisation ensures that only the first call incurs any overheads from the wrapping; after that, calls are made directly to the target function.