Changelog

4.3.2

  • Speed up receiving UDP with the Linux kernel network stack by using generic receive offload (GRO).

  • Update Boost version in wheels to 1.85.

  • Fix compatibility with numpy 2.0.

  • Change the unit tests to allocate TCP and UDP ports dynamically.

  • Add a series of tutorials to the manual.

  • Add a missing type annotation for spead2.ThreadPool.set_affinity().

  • Add a Jenkins configuration to test the ibverbs support on a SARAO-internal Jenkins server.

  • Bump versions of dependencies used in CI.

4.3.1

  • Switch from netifaces to netifaces2 for testing.

  • Update coverallsapp/github-action to v2.2.3.

  • Fix the type annotation for spead2.recv.StreamConfig to allow explicit_start to be passed as a constructor argument.

4.3.0

  • Add ability to override the transmission rate for individual heaps.

  • Fix missing type annotation on the substream_index argument to send_heap().

4.2.0

  • Significantly speed up transmission when using the Linux kernel networking stack, by using generic segmentation offload.

  • Speed up transmission for small packets (20% in some cases).

  • Make async_send_heap() and async_send_heaps() accept completion tokens. This allows for code like stream.async_send_heap(heap, boost::asio::use_future);

  • Add C++ support to iterate over ring_stream and ringbuffer using a range-based for loop.

  • Add support for the prefer_huge argument to the MmapAllocator constructor in Python.

  • Make the allocator argument to MemoryPool optional in Python.

  • Allow indexing a HeapReferenceList with a slice to create a new HeapReferenceList with a subset of the heaps.

  • Add ringbuffer to the C++ documentation.

  • Eliminate use of the deprecated boost::asio::null_buffers.

  • Update Boost version in wheels to 1.84.

  • Update rdma-core version in manylinux wheels to 49.0.

  • Make various updates to the Github Actions infrastructure.

4.1.1

  • Add AVX and AVX-512 implementations for non-temporal memory copy. While this is transparent to the API, the Meson option names have changed to allow specific instruction sets to be disabled if necessary.

4.1.0

4.0.2

Note that an oversight lead to some of the changes between 4.0.0b1 and 4.0.0 being omitted from 4.0.1. They are restored in 4.0.2.

  • Fix type annotations for spead2.send.UdpStream and spead2.send.asyncio.UdpStream.

  • Add more documentation for developers.

  • Remove an old Makefile.am that should have been removed in 4.0.0.

  • Remove mocking of spead2 in readthedocs build.

  • Change .. code:: to .. code-block:: in documentation.

  • Simplify the implementation of thread_pool_wrapper and buffer_reader in Python binding code.

  • Directly use pointer-to-member-functions in Python binding code.

  • Test against numpy 1.26.0 (instead of 1.26.0rc1) on Python 3.12.

4.0.1

  • Restore dependency on numpy, which was accidentally removed in 4.0.0.

  • Change the test_numba extra to test-numba to normalise it in accordance with PEP 685.

4.0.0

This release makes major changes to the build system, and removes deprecated features. See Migrating to version 4 for more detailed information about upgrading and the deprecation removals.

Most of the changes are listed under 4.0.0b1 below. Since then, the following changes have been made:

  • Improve detection of gdrcopy by using the CUDA compiler to compile the test code.

  • Remove ninja from build-system.requires. If you have ninja installed on the system, that will be used for the Python install rather than downloading it.

  • Make miscellaneous improvements to the build system.

  • Remove an unused file (.ci/ccache-path.sh).

  • Work around a pytest bug to prevent tests running out of file descriptors (particularly on MacOS, which has a lower default limit).

  • Add wheels for MacOS (Intel and Apple Silicon).

  • Document that Meson must be at least 1.2.

  • Make source paths in .debug files more usable (relative to 4.0.0b1).

  • Remove .pyi file entries for the functionality removed in 4.0.

4.0.0b1

This release makes major changes to the build system, and removes deprecated features. See Migrating to version 4 for more detailed information about upgrading and the deprecation removals.

  • Replace the build system with Meson.

  • Update the C++ code to use C++17.

  • No longer link against boost_system, and require Boost 1.69+.

  • Remove generated code from release tarballs; some Python packages are now required at build time to build the C++ bindings.

  • Fix an uninitialised variable that could cause a segmentation fault when using TCP to send and the initial connection failed.

  • Fix a large number of compiler warnings that showed up after switching build systems (mainly related to unused function parameters and signed/unsigned comparisons).

  • Fix the debug logging option so logging from inline functions in headers will also do debug logging without the user needing to make preprocessor defines.

  • Fix spead2_bench.py so that it functions when ibverbs support is not compiled in.

  • Remove the need for boost_program_options to be installed to be able to install the Python bindings from source.

  • Produce binary wheels for aarch64.

  • Produce wheels for Python 3.12.

  • Make numba and scipy optional for running tests (some tests will be skipped if they are not present).

  • Update the libpcap embedded in the wheels to 1.10.4.

  • Update the Boost version used to build wheels to 1.83.

  • Update the rdma-core version used to build wheels to 47.0.

  • Update the pybind11 build dependency to 2.11.1.

  • Replace flake8 with ruff for linting.

  • Remove the spead2/common_bind.h header, which was unused.

  • Remove the SPEAD2_DEPRECATED macro.

  • Remove build-time dependencies from requirements.txt.

  • Update the .pyi files to use more modern syntax e.g., PEP 585, PEP 604, PEP 613.

  • Replace references to nv_peer_mem with nvidia-peermem.

  • Increase TTL of gpudirect_example to 4.

3.13.0

  • Reformat the Python codebase using black and isort.

  • Add pre-commit configuration.

  • On i386, check for SSE2 support at runtime rather than configure time.

  • Free readers only when the stream is destroyed. This fixes a bug that caused the Python API to be accessed without the GIL when using add_buffer_reader().

  • Improve unit tests by explicitly closing TCP sockets, to avoid ResourceWarning when testing with python -X dev.

  • Remove wheel from build-system.requires.

3.12.0

  • Add support for Chunking stream groups to assemble chunks in parallel.

  • Simplify the way receive streams shut down. Users should not notice any change, but custom reader implementations will need to be updated.

  • Update test_async_flush() and test_async_flush_fail() to keep handles to async tasks, to prevent them being garbage collected too early.

  • Fix a bug where copying a spead2::recv::stream_config would not deep copy the names of custom statistics, and so any statistics added to the copy would also affect the original, and there were also potential race conditions if a stream config was modified while holding stream statistics.

  • Fix a bug (caused by the bug above) where passing a spead2::recv::stream_config to construct a spead2::recv::chunk_stream would modify the config. Passing the same config to construct two chunk streams would fail with an error.

  • Fix the type annotation for the ChunkRingStream constructor: the parameter name for chunk_stream_config was incorrect.

  • Fix universal binary builds on MacOS (this was preventing Python 3.11 builds from succeeding).

  • Fix spead2_bench.py, which has failed to run at all for some time (possibly since 3.0).

  • Avoid including Boost dynamic symbols in the Python module (helps reduce binary size).

  • Strip static symbols out of the Python wheels (reduces size).

  • Build Python wheels with link-time optimisation (small performance improvement).

  • Python 3.8 is now the minimum version.

3.11.1

  • Fix a packaging issue that meant automake and similar tools were required to compile (since 3.10).

3.11.0

  • The chunking receiver is no longer experimental.

  • The place callback for the chunking receiver can now provide extra data to be written to the chunk.

3.10.0

  • Support pcap dumps that use the SLL format.

  • Support a user-defined filter in the pcap file reader.

  • Add experimental support for building a shared library.

  • Assorted documentation updates

    • The SPEAD specification is now stored in the repository (the upstream link is broken).

    • Build PDFs on readthedocs.

    • Update the tuning documentation.

3.9.1

3.9.0

  • Added substreams to spead2.recv.StreamConfig to improve handling of interleaved heaps from multiple senders.

  • Add libdivide to the dependencies.

3.8.0

  • Drop support for Python 3.6, which has reached end-of-life.

  • Test against Python 3.10 in Github Actions.

  • Improve the accuracy of the rate limiter. Previously it could send slightly too fast due to rounding sleep times to whole numbers of nanoseconds.

  • Eliminate dependence on distutils, which is deprecated in Python 3.10 (#175).

3.7.0

  • Add spead2.send.GroupMode.SERIAL.

  • Add spead2.send.HeapReferenceList.

  • Speed up C++ unit tests.

  • Fix some spurious output in the statistics report from spead2_recv.py (introduced in 3.5.0).

  • Fix the help message from spead2_net_raw to have the right name for the program.

  • Update to latest version of pybind11.

3.6.0

  • Allow a ringbuffer to be stopped only once the last producer has indicated completion, rather than the first.

  • Change ChunkRingStream so that stops received from the network only shut down a shared ringbuffer once all the streams have stopped. A user call to stop will still stop the ringbuffer immediately.

  • ChunkRingbuffer.stop() now returns a boolean to indicate whether this is the first time the ringbuffer was stopped.

3.5.0

  • Add support for Custom statistics.

  • Change the allocate and ready callbacks on spead2::recv::chunk_stream to take a pointer to the batch statistics. This is a backwards-incompatible change (keep in mind that chunking receive is still experimental). Code that uses spead2::recv::chunk_ring_stream is unaffected.

  • Change the design of deleters for spead2::memory_allocator. Code that calls get_user or get_deleter on a pointer allocated by spead2 may now get a nullptr back. Code that uses a custom memory allocator and that calls these functions on pointers allocated by that allocator should continue to work.

  • Allow a ready callback to be used together with spead2::recv::chunk_ring_stream, to finish preparation of a chunk before it pushed to the ringbuffer.

  • In Python, avoid copying immediate items when given as 0-d arrays with dtype >u8. This makes it practical to pre-define heaps and later update their values rather than creating new heap objects.

  • Make spead2.send.Stream, spead2.send.SyncStream and spead2.send.asyncio.AsyncStream available for type annotations.

  • Fix an occasional segfault when stopping a spead2.recv.ChunkRingStream.

3.4.0

3.3.2

3.3.1

  • Convert spead2_net_raw to a C++ file so that it gets the same compiler flags as everything else.

  • Migrate from Travis CI to Github Actions.

  • Fix some warnings generated by Clang.

  • Fix some test failures with PyPy.

3.3.0

  • Add spead2_net_raw tool.

  • Eliminate some compiler warnings about unused parameters.

  • Update build process to use pypa-build and setuptools_scm.

  • Update to pybind11 2.6.2.

3.2.2

  • Use python3 instead of python to invoke Python (so that it works even on systems where python is absent or is Python 2).

  • Work around a bug that prevented compilation on Boost 1.76.

3.2.1

  • Update type annotations to use numpy.typing.DTypeLike for dtype arguments, to prevent false warnings from mypy.

3.2.0

  • Add spead2::recv::heap::get_payload() to allow the payload pointer to be retrieved from a complete heap.

  • Make the ibverbs sender compatible with PeerDirect.

  • Add examples programs showing integration with gdrcopy and PeerDirect.

  • Always use SFENCE at end of memcpy_nontemporal() so that it is appropriate for use with gdrcopy.

  • Fix a memory leak when receiving with ibverbs.

3.1.3

  • Fix installation of header files: some newer headers were not being installed, breaking builds for C++ projects.

3.1.2

  • Fix a use-after-free bug that could cause a crash when freeing a send stream.

  • Improve send performance by eliminating a memory allocation from packet generation.

3.1.1

  • Set IBV_ACCESS_RELAXED_ORDERING flag on ibverbs memory regions. This reduces packet loss in some circumstances (observed on Epyc 2 system with lots of memory traffic).

3.1.0

  • Add send_heaps() and async_send_heaps() to send groups of heaps with interleaved packets.

  • Upgrade to pybind11 2.6.0, which contains a workaround for a bug in CPython 3.9.0.

3.0.1

  • Bring the type stubs up to date.

  • Fix a typo in the documentation.

3.0.0

Version 3.0 contains a number of breaking API changes. For information on updating your existing code, refer to Migrating to version 3.

The ibverbs acceleration has been substantially modified to use a newer version of rdma-core. It will no longer compile against versions of MLNX-OFED prior to 5.0. Compiled code (such as Python wheels) will still run against old versions of MLNX-OFED, but extension features such as multi-packet receive queues and packet timestamps will not work, and nor will mcdump. It is recommended that if you are using ibverbs acceleration with older MLNX-OFED drivers that you stick with spead2 2.x until you’re able to upgrade the drivers and spead2 simultaneously.

  • Support multiple “substreams” in a send stream (see Substreams).

  • Reduce overhead for dealing with incomplete heaps.

  • Allow ibverbs senders to register memory regions for zero-copy transmission.

  • Add C++ preprocessor defines for the version number.

  • Use IP/UDP checksum offloading for sending with ibverbs (improves performance and also adds UDP checksum which is otherwise omitted).

  • Add wheels for Python 3.9.

  • Drop support for Python 3.5, which is end-of-life.

  • Change code examples to use standard SPEAD rather than PySPEAD bug compatibility.

  • Change spead2::send::streambuf_stream so that when the streambuf only partially writes a packet, the partial byte count is included in the count returned to the callback.

  • spead2::send::stream::flush() now only blocks until the previously enqueued heaps are completed. Another thread that keeps adding heaps would previously have prevented it from returning.

  • Partially rewrite the sending infrastructure, resulting in performance improvements, in some cases of over 10%.

  • Setting a buffer size of 0 for a UdpIbvStream now uses the default buffer size, instead of a 1-packet buffer.

  • Fix spead2_bench.py ignoring the --send-affinity option.

  • Add --verify option to spead2_send and spead2_recv to aid in testing the code. To support this, spead2_send was modified so that each in-flight heap uses different memory, which may reduce performance (due to less cache re-use) even when the option is not given.

  • Miscellaneous performance improvements.

  • Support hardware send rate limiting when using ibverbs (disabled by default).

  • Discover libibverbs and pcap using pkg-config where possible.

  • Make configure print out the configuration that will be compiled.

  • Update the Python wheels to use manylinux2014. This uses a newer compiler (potentially giving better performance) and supports sendmmsg().

  • A number of deprecated functions have been removed.

  • Avoid ibverbs code creating a send queue for receiver or vice versa.

  • Rename slave option to spead2_bench to agent.

Compared to 3.0.0b2 there is a critical bug fix for a race condition in the send code.

3.0.0b2

Version 3.0 contains a number of breaking API changes. For information on updating your existing code, refer to Migrating to version 3.

Other changes:

  • Support multiple “substreams” in a send stream (see Substreams).

  • Reduce overhead for dealing with incomplete heaps.

  • Allow ibverbs senders to register memory regions for zero-copy transmission.

  • Add C++ preprocessor defines for the version number.

  • Use IP/UDP checksum offloading for sending with ibverbs (improves performance and also adds UDP checksum which is otherwise omitted).

  • Drop support for Python 3.5, which is end-of-life.

  • Change code examples to use standard SPEAD rather than PySPEAD bug compatibility.

  • Change spead2::send::streambuf_stream so that when the streambuf only partially writes a packet, the partial byte count is included in the count returned to the callback.

  • spead2::send::stream::flush() now only blocks until the previously enqueued heaps are completed. Another thread that keeps adding heaps would previously have prevented it from returning.

  • Partially rewrite the sending infrastructure, resulting in performance improvements, in some cases of over 10%.

  • Setting a buffer size of 0 for a UdpIbvStream now uses the default buffer size, instead of a 1-packet buffer.

  • Fix spead2_bench.py ignoring the --send-affinity option.

  • The hardware rate limiting introduced in 3.0.0b1 is now disabled by default, as it proved to be significantly less accurate than the software rate limiter in some cases. The interface has also been changed from a boolean to an enum (with the default being AUTO) so that it can later be re-enabled under circumstances where it is known to work well, while still allowing it to be explicitly enabled or disabled.

  • Add --verify option to spead2_send and spead2_recv to aid in testing the code. To support this, spead2_send was modified so that each in-flight heap uses different memory, which may reduce performance (due to less cache re-use) even when the option is not given.

  • Miscellaneous performance improvements.

Additionally, refer to the changes for 3.0.0b1 below.

3.0.0b1

The ibverbs acceleration has been substantially modified to use a newer version of rdma-core. It will no longer compile against versions of MLNX-OFED prior to 5.0. Compiled code (such as Python wheels) will still run against old versions of MLNX-OFED, but extension features such as multi-packet receive queues and packet timestamps will not work. It is recommended that if you are using ibverbs acceleration with older MLNX-OFED drivers that you stick with spead2 2.x until you’re able to upgrade the drivers and spead2 simultaneously.

Other changes:

  • Support hardware send rate limiting when using ibverbs.

  • Discover libibverbs and pcap using pkg-config where possible.

  • Make configure print out the configuration that will be compiled.

  • Update the Python wheels to use manylinux2014. This uses a newer compiler (potentially giving better performance) and supports sendmmsg().

  • Add wheels for Python 3.9.

  • A number of deprecated functions have been removed.

  • Avoid ibverbs code creating a send queue for receiver or vice versa.

  • Rename slave option to spead2_bench to agent.

2.1.2

  • Make verbs acceleration work when run against MLNX OFED 5.x, including with Python wheels. Note that it will not use multi-packet receive queues, so receive performance may still be better on MLNX OFED 4.9.

2.1.1

  • Update pybind to 2.5.0.

  • Fix compilation against latest rdma-core.

  • Some documentation cleanup.

2.1.0

  • Support unicast receive with ibverbs acceleration (including in mcdump).

  • Fix spead2_recv listening only on loopback when given just a port number.

  • Support unicast addresses in a few APIs that previously only accepted multicast addresses; in most cases the unicast address must match the interface address.

  • Add missing <map> include to <spead2/recv_heap.h>.

  • Show the values of immediate items in spead2_recv.

  • Fix occasional crash when using thread pool with more than one thread together with ibverbs.

  • Fix bug in mcdump causing it to hang if the arguments couldn’t be parsed (only happened when capturing to file).

  • Fix spead2_recv reporting statistics that may miss out the last batch of packets.

2.0.2

  • Log warnings on some internal errors (that hopefully never happen).

  • Include wheels for Python 3.8.

  • Build debug symbols for binary wheels (in a separate tarball on Github).

2.0.1

  • Fix race condition in TCP receiver (#78).

  • Update vendored pybind11 to 2.4.2.

2.0.0

  • Drop support for Python 2.

  • Drop support for Python 3.4.

  • Drop support for trollius.

  • Drop support for netmap.

  • Avoid creating some cyclic references. These were not memory leaks, but prevented CPython from freeing objects as soon as it might have.

  • Update vendored pybind11 to 2.4.1.

1.14.0

1.13.1

  • Raise ValueError on a dtype that has zero itemsize (#37).

  • Change exception when dtype has embedded objects from TypeError to ValueError for consistency

  • Remove duplicated socket handle in UDP receiver (#67).

  • Make max_poll argument to spead2.send.UdpIbvStream actually have an effect (#55).

  • Correctly report EOF errors in spead2::send::streambuf_stream.

  • Wrap implicitly computed heap cnts to the number of available bits (#3). Previously behaviour was undefined.

  • Some header files were not installed by make install (#72).

1.13.0

  • Significant performance improvements to send code (in some cases an order of magnitude improvement).

  • Add --max-heap option to spead2_send and spead2_send.py to control the depth of the send queue.

  • Change the meaning of the --heaps option in spead2_bench and spead2_bench.py: it now also controls the depth of the sending queue.

  • Fix a bug in send rate limiting that could allow the target rate to be exceeded under some conditions.

  • Remove --threads option from C++ spead2_send, as the new optimised implementation isn’t thread-safe.

  • Disable the test_numpy_large test on macOS, which was causing frequent failures on TravisCI due to dropped packets.

1.12.0

  • Provide manylinux2010 wheels.

  • Dynamically link to libibverbs and librdmacm on demand. This allows binaries (particularly wheels) to support verbs acceleration but still work on systems without these libraries installed.

  • Support for Boost 1.70. Unfortunately Boost 1.70 removes the ability to query the io_service from a socket, so constructors that take a socket but no io_service are omitted when compiling with Boost 1.70 or newer.

  • Fix some compiler warnings from GCC 8.

1.11.4

  • Rework the locking internals of spead2::recv::stream so that a full ringbuffer doesn’t block new readers from being added. This changes the interfaces between spead2::recv::reader and spead2::recv::stream_base, but since users generally don’t deal with that interface the major version hasn’t been incremented.

  • Fix a spurious log message if an in-process receiver is manually stopped.

  • Fix an intermittent unit test failure due to timing.

1.11.3

  • Undo the optimisation of using a single flow steering rule to cover multiple multicast groups (see #11).

1.11.2

  • Fix -c option to mcdump.

  • Fix a missing #include that could be exposed by including headers in a particular order.

  • Make spead2::recv::heap’s move constructor and move assignment operator noexcept.

  • Add a long_description to the Python metadata.

1.11.1

  • Update type stubs for new features in 1.11.0.

1.11.0

  • Add spead2.recv.Stream.allow_unsized_heaps to support rejecting packets without a heap length.

  • Add extended custom memcpy support (C++ only) for scattering data from packets.

1.10.1

  • Use ibverbs multi-packet receive queues automatically when available (supported by mlx5 driver).

  • Automatically reduce buffer size for verbs receiver to match hardware limits (fixed #64).

  • Gracefully handle Ctrl-C in spead2_recv and print statistics.

  • Add typing stub files to assist checking with Mypy.

  • Give a name to the argument of spead2.recv.Stream.add_inproc_reader().

  • Fix Python binding for one of the UDP reader overloads that takes an existing socket. This was a deprecated overload.

  • Add a unit test for ibverbs support. It’s not run by default because it needs specific hardware.

1.10.0

  • Accelerate per-packet processing, particularly when max_heaps is large.

  • Accelerate per-heap processing, particularly for heaps with few items.

  • Add a fast path for single-packet heaps.

  • Improve performance of the pcap reader by working on batches of packets.

  • Provide access to ringbuffer size and capacity for diagnostics.

  • Add extra fields to spead2.recv.StreamStats.

  • Add support for pcap files to the C++ version of spead2_recv.

  • Update the vendored pybind11 to 2.2.4 (fixes some warnings on Python 3.7).

  • Deprecate netmap support in documentation.

1.9.2

  • autotools are no longer required to install the C++ build (when installing from a release tarball).

1.9.1

1.9.0

  • Add support for TCP/IP (contributed by Rodrigo Tobar).

  • Changed command-line options for spead2_send/spead2_recv: --ibv and --netmap are now boolean flags, and the interface address is set with --bind.

  • Added option to specify interface address for spead2::send::udp_stream even when not using the multicast constructors.

  • Constructors that take an existing socket now expect the user to set all socket options. The old versions that take a socket buffer size are deprecated. Note that the behaviour of spead2::send::udp_stream with a socket has changed: if no buffer size is given, it is left at the OS default, rather than applying the spead2 default.

  • Fix a bug causing undefined behaviour if a send class is destroyed while there is still data in flight.

Version 1.8.0

Version 1.7.2

  • Add progress reports to mcdump

  • Add ability to pass - as filename to mcdump to skip file writing.

  • Add --count option to mcdump

Version 1.7.1

There are no code changes, but this release fixes a packaging error in 1.7.0 that prevented the asyncio integration from being included.

Version 1.7.0

  • Support for pcap files. Files passed to spead2_recv.py are now assumed to be pcap files, rather than raw concatenated packets.

  • Only log warnings about the ringbuffer being full if at least one stream reader is lossy (indicated by a new virtual member function in spead2::recv::Reader).

Version 1.6.0

  • Change spead2_send.py and spead2_send to interpret the --rate option as Gb/s and not Gib/s.

  • Change send rate limiting to bound the rate at which we catch up if we fall behind. This is controlled by a new attribute of StreamConfig.

  • Add report at end of spead2_send.py and spead2_send on the actual number of bytes sent and achieved rate.

  • Fix a race condition where the stream statistics might only be updated after the stream ended (which lead to unit test failures in some cases).

Version 1.5.2

  • Report statistics when spead2_recv.py is stopped by SIGINT.

  • Add –ttl option to spead2_send.py and spead2_send.

Version 1.5.1

  • Explicitly set UDP checksum to 0 in IBV sender, instead of leaving arbitrary values.

  • Improved documentation of asyncio support.

Version 1.5.0

  • Support for asyncio in Python 3. For each trollius module there is now an equivalent asyncio module. The installed utilities use asyncio on Python 3.4+.

  • Add spead2.recv.Stream.stop_on_stop_item to allow a stream to keep receiving after a stop item is received.

  • Switch shutdown code to use atexit instead of a capsule destructor, to support PyPy.

  • Test PyPy support with Travis.

Version 1.4.0

  • Remove --bind option to spead2_recv.py and spead2_recv. Instead, use host:port as the source. This allows subscribing to multiple multicast groups.

  • Improved access to information about incomplete heaps (spead2.recv.IncompleteHeap type).

  • Add MemoryPool.warn_on_empty control.

  • Add warning when a stream ringbuffer is full.

  • Add statistics to streams.

  • Fix spead2_send.py to send a stop heap when using --heaps. It was acccidentally broken in 1.2.0.

  • Add support for packet timestamping in mcdump.

  • Return the previous logging function from spead2::set_log_function().

  • Make Python logging from C++ code asynchronous, to avoid blocking the thread pool on the GIL.

  • Upgrade to pybind11 2.2.1 internally.

  • Some fixes for PyPy support.

Version 1.3.2

  • Fix segfault in shutdown for spead2_recv.py (fixes #56).

  • Fix for TypeError in Python 3.6 when reading fields that aren’t aligned to byte boundaries.

  • Include binary wheels in releases.

Version 1.3.1

Version 1.3.0

  • Rewrite the Python wrapping using pybind11. This should not cause any compatibility problems, unless you’re using the spead2/py_*.h headers.

  • Allow passing std::shared_ptr to constructors that take a thread pool, with the constructed object holding a reference.

  • Prevent constructing a spead2.recv.Stream with max_heaps=0 (fixes #54).

Version 1.2.2

  • Fix rate limiting causing longer sleeps than necessary (fixes #53).

Version 1.2.1

  • Disable LTO by default and require the user to opt in, because even if the compiler supports it, linking can still fail (fixes #51).

Version 1.2.0

  • Support multiple endpoints for one udp_ibv_reader (fixes #48).

  • Fix compilation on OS X 10.9 (fixes #49)

  • Fix spead2::ringbuffer<T>::emplace() and spead2::ringbuffer<T>::try_emplace()

  • Improved error messages when passing invalid arguments to mcdump

Version 1.1.2

  • Only log descriptor replacement if it actually replaces an existing name or ID (regression in 1.1.1).

  • Fix build on ARM where compiling against asio requires linking against pthread.

  • Updated and expanded performance tuning guide.

Version 1.1.1

  • Report the item name in exception for “too few elements for shape” errors

  • Overhaul of rules for handling item descriptors that change the name or ID of an item. This prevents stale items from hanging around when the sender changes the name of an item but keeps the same ID, which can cause unrelated errors on the receiver if the shape also changes.

Version 1.1.0

  • Allow heap cnt to be set explicitly by sender, and the automatic heap cnt sequence to be specified as a start value and step.

Version 1.0.1

  • Fix exceptions to include more information about the source of the failure

  • Add mcdump tool

Version 1.0.0

  • The C++ API installation has been changed to use autoconf and automake. As a result, it is possible to run make install and get the static library, headers, and tools installed.

  • The directory structure has changed. The spead2_* tools are now installed, example code is now in the examples directory, and the headers have moved to include/spead2.

  • Add support for sending data using libibverbs API (previously only supported for receiving)

  • Fix async_send_heap (in Python) to return a future instead of being a coroutine: this fixes a problem with undefined ordering in the trollius example.

  • Made sending streams polymorphic, with abstract base class spead2::send::stream, to simplify writing generic code that can operate on any type of stream. This will break code that depended on the old template class of the same name, which has been renamed to spead2::send::stream_impl.

  • Add --memcpy-nt to spead2_recv.py and spead2_bench.py

  • Multicast support in spead2_bench.py and spead2_bench

  • Changes to the algorithm for spead2_bench.py and spead2_bench: it now starts by computing the maximum send speed, and then either reporting that this is the limiting factor, or using it to start the binary search for the receive speed. It is also stricter about lost heaps.

  • Some internal refactoring of code for dealing with raw packets, so that it is shared between the netmap and ibv readers.

  • Report function name that failed in semaphore system_error exceptions.

  • Make the unit tests pass on OS X (now tested on travis-ci.org)

Version 0.10.4

  • Refactor some of the Boost.Python glue code to make it possible to reuse parts of it in writing new Python extensions that use the C++ spead2 API.

Version 0.10.3

  • Suppress “operation aborted” warnings from UDP reader when using the API to stop a stream (introduced in 0.10.0).

  • Improved elimination of duplicate item pointers, removing them as they’re received rather than when freezing a live heap (fixes #46).

  • Use hex for reporting item IDs in log messages

  • Fix reading from closed file descriptor after stream.stop() (fixes #42)

  • Fix segmentation fault when using ibverbs but trying to bind to a non-RDMA device network interface (fixes #45)

Version 0.10.2

  • Fix a performance problem when a heap contains many packets and every packet contains item pointers. The performance was quadratic instead of linear.

Version 0.10.1

  • Fixed a bug in registering add_udp_ibv_reader in Python, which broke spead2_recv.py, and possibly any other code using this API.

  • Fixed spead2_recv.py ignoring --ibv-max-poll option

Version 0.10.0

  • Added support for libibverbs for improved performance in both Python and C++.

  • Avoid per-packet shared_ptr reference counting, accidentally introduced in 0.9.0, which caused a small performance regression. This is unfortunately a breaking change to the interface for implementing custom memory allocators.

Version 0.9.1

  • Fix using a MemoryPool with a thread pool and low water mark (regression in 0.9.0).

Version 0.9.0

  • Add support for custom memory allocators.

Version 0.8.2

  • Ensure correct operation when loop=None is passed explicitly to trollius stream constructors, for consistency with functions that have it as a keyword parameter.

Version 0.8.1

  • Suppress recvmmsg: resource temporarily unavailable warnings (fixes #43)

Version 0.8.0

  • Extend MemoryPool to allow a background thread to replenish the pool when it gets low.

  • Extend ThreadPool to allow the user to pin the threads to specific CPU cores (on glibc).

Version 0.7.1

  • Fix ring_stream destructor to not deadlock (fixes #41)

Version 0.7.0

  • Change handling of incomplete heaps (fixes #39). Previously, incomplete heaps were only abandoned once there were more than max_heaps of them. Now, they are abandoned once max_heaps more heaps are seen, even if those heaps were complete. This causes the warnings for incomplete heaps to appear closer to the time they arrived, and also has some extremely small performance advantages due to changes in the implementation.

  • backwards-incompatible change: remove set_max_heaps(). It was not previously documented, so hopefully is not being used. It could not be efficiently supported with the design changes above.

  • Add spead2.recv.Stream.set_memcpy() to control non-temporal caching hints.

  • Fix C++ version of spead2_bench to actually use the memory pool

  • Reduce memory usage in spead2_bench (C++ version)

Version 0.6.3

  • Partially fix #40: set_max_heaps() and set_memory_pool() will no longer deadlock if called on a stream that has already had a reader added and is receiving data.

Version 0.6.2

  • Add a fast path for integer items that exactly fit in an immediate.

  • Optimise Python code by replacing np.product with a pure Python implementation.

Version 0.6.1

  • Filter out duplicate items from a heap. It is undefined which of a set of duplicates will be retained (it was already undefined for spead2.ItemGroup).

Version 0.6.0

  • Changed item versioning on receive to increment version number on each update rather that setting to heap id. This is more robust to using a single item or item group with multiple streams, and most closely matches the send path.

  • Made the protocol enums from the C++ library available in the Python library as well.

  • Added functions to create stream start items (send) and detect them (recv).

Version 0.5.0

  • Added friendlier support for multicast. When a multicast address is passed to add_udp_reader(), the socket will automatically join the multicast group and set SO_REUSEADDR so that multiple sockets can consume from the same stream. There are also new constructors and methods to give explicit control over the TTL (send) and interface (send and receive), including support for IPv6.

Version 0.4.7

  • Added in-memory mode to the C++ version of spead2_bench, to measure the packet handling speed independently of the lossy networking code

  • Optimization to duplicate packet checks. This makes a substantial performance improvement when using small (e.g. 512 byte) packets and large heaps.

Version 0.4.6

  • Fix a data corruption (use-after-free) bug on send side when data is being sent faster than the socket can handle it.

Version 0.4.5

  • Fix bug causing some log messages to be remapped to DEBUG level

Version 0.4.4

  • Increase log level for packet rejection from DEBUG to INFO

  • Some minor optimisations

Version 0.4.3

  • Handle heaps that have out-of-range item offsets without crashing (#32)

  • Fix handling of heaps without heap length headers

  • spead2.send.UdpStream.send_heap() now correctly raises IOError if the heap is rejected due to being full, or if there was an OS-level error in sending the heap.

  • Fix spead2.send.trollius.UdpStream.async_send_heap() for the case where the last sent heap failed.

  • Use eventfd(2) for semaphores on Linux, which makes a very small improvement in ringbuffer performance.

  • Prevent messages about descriptor replacements for descriptor reissues with no change.

  • Fix a use-after-free bug (affecting Python only).

  • Throw OverflowError on out-of-range UDP port number, instead of wrapping.

Version 0.4.2

  • Fix compilation on systems without glibc

  • Fix test suite for non-Linux systems

  • Add spead2.send.trollius.UdpStream.async_flush()

Version 0.4.1

  • Add C++ version of spead2_recv, a more fully-featured alternative to test_recv

  • backwards-incompatible change: Add ring_heaps parameter to ring_stream constructor. Code that specifies the contiguous_only parameter will need to be modified since the position has changed. Python code is unaffected.

  • Increased the default for ring_heaps from 2 (previously hardcoded) to 4 to improve throughput for small heaps.

  • Add support for user to provide the socket for UDP communications. This allows socket options to be set by the user, for example, to configure multicast.

  • Force numpy>=1.9.2 to avoid a numpy [bug](https://github.com/numpy/numpy/issues/5356).

  • Add experimental support for receiving packets via netmap

  • Improved receive performance on Linux, particularly for small packets, using [recvmmsg](http://linux.die.net/man/2/recvmmsg).

Version 0.4.0

  • Enforce ASCII encoding on descriptor fields.

  • Warn if a heap is dropped due to being incomplete.

  • Add –ring option to C++ spead2_bench to test ringbuffer performance.

  • Reading from a memory buffer (e.g. with add_buffer_reader()) is now reliable, instead of dropping heaps if the consumer doesn’t keep up (heaps can still be dropped if packets extracted from the buffer are out-of-order, but it is deterministic).

  • The receive ringbuffer now has a fixed size (2), and pushes are blocking. The result is lower memory usage, and it is no longer necessary to pass a large max_heaps value to deal with the consumer not always keeping up. Instead, it may be necessary to increase the socket buffer size.

  • backwards-incompatible change: Calling spead2::recv::ring_stream::stop() now discards remaining partial heaps instead of adding them to the ringbuffer. This only affects the C++ API, because the Python API does not provide any access to partial heaps anyway.

  • backwards-incompatible change: A heap with a stop flag is swallowed rather than passed to heap_ready() (see issue [#29](https://github.com/ska-sa/spead2/issues/29)).

Version 0.3.0

This release contains a number of backwards-incompatible changes in the Python bindings, although most uses will probably not notice:

  • When a received character array is returned as a string, it is now of type str (previously it was unicode in Python 2).

  • An array of characters with a numpy descriptor with type S1 will no longer automatically be turned back into a string. Only using a format of [(‘c’, 8)] will do so.

  • The c format code may now only be used with a length of 8.

  • When sending, values will now always be converted to a numpy array first, even if this isn’t the final representation that will be put on the network. This may lead to some subtle changes in behaviour.

  • The BUG_COMPAT_NO_SCALAR_NUMPY introduced in 0.2.2 has been removed. Now, specifying an old-style format will always use that format at the protocol level, rather than replacing it with a numpy descriptor.

There are also some other bug-fixes and improvements:

  • Fix incorrect warnings about send buffer size.

  • Added –descriptors option to spead2_recv.py.

  • The dtype argument to spead2.ItemGroup.add_item() is now optional, removing the need to specify dtype=None when passing a format.

Version 0.2.2

  • Workaround for a PySPEAD bug that would cause PySPEAD to fail if sent a simple scalar value. The user must still specify scalars with a format rather than a dtype to make things work.

Version 0.2.1

  • Fix compilation on OS X again. The extension binary will be slightly larger as a result, but still much smaller than before 0.2.0.

Version 0.2.0

  • backwards-incompatible change: for sending, the heap count is now tracked internally by the stream, rather than an attribute of the heap. This affects both C++ and Python bindings, although Python code that always uses HeapGenerator rather than directly creating heaps will not be affected.

  • The HeapGenerator is extended to allow items to be added to an existing heap and to give finer control over whether descriptors and/or values are put in the heap.

  • Fixes a bug that caused some values to be cast to non-native endian.

  • Added overloaded equality tests on Flavour objects.

  • Strip the extension binary to massively reduce its size

Version 0.1.2

  • Coerce values to int for legacy ‘u’ and ‘i’ fields

  • Fix flavour selection in example code

Version 0.1.1

  • Fixes to support OS X

Version 0.1.0

  • First public release