Changelog¶
1.12.0
- Provide manylinux2010 wheels.
- Dynamically link to libibverbs and librdmacm on demand. This allows binaries (particularly wheels) to support verbs acceleration but still work on systems without these libraries installed.
- Support for Boost 1.70. Unfortunately Boost 1.70 removes the ability to query the io_service from a socket, so constructors that take a socket but no io_service are omitted when compiling with Boost 1.70 or newer.
- Fix some compiler warnings from GCC 8.
1.11.4
- Rework the locking internals of
spead2::recv::stream
so that a full ringbuffer doesn’t block new readers from being added. This changes the interfaces betweenspead2::recv::reader
andspead2::recv::stream_base
, but since users generally don’t deal with that interface the major version hasn’t been incremented. - Fix a spurious log message if an in-process receiver is manually stopped.
- Fix an intermittent unit test failure due to timing.
1.11.3
- Undo the optimisation of using a single flow steering rule to cover multiple multicast groups (see #11).
1.11.2
- Fix
-c
option to mcdump. - Fix a missing
#include
that could be exposed by including headers in a particular order. - Make
spead2::recv::heap
’s move constructor and move assignment operatornoexcept
. - Add a long_description to the Python metadata.
1.11.1
- Update type stubs for new features in 1.11.0.
1.11.0
- Add
spead2.recv.Stream.allow_unsized_heaps
to support rejecting packets without a heap length. - Add extended custom memcpy support (C++ only) for scattering data from packets.
1.10.1
- Use ibverbs multi-packet receive queues automatically when available (supported by mlx5 driver).
- Automatically reduce buffer size for verbs receiver to match hardware limits (fixed #64).
- Gracefully handle Ctrl-C in spead2_recv and print statistics.
- Add typing stub files to assist checking with Mypy.
- Give a name to the argument of
spead2.recv.Stream.add_inproc_reader()
. - Fix Python binding for one of the UDP reader overloads that takes an existing socket. This was a deprecated overload.
- Add a unit test for ibverbs support. It’s not run by default because it needs specific hardware.
1.10.0
- Accelerate per-packet processing, particularly when max_heaps is large.
- Accelerate per-heap processing, particularly for heaps with few items.
- Add a fast path for single-packet heaps.
- Improve performance of the pcap reader by working on batches of packets.
- Provide access to ringbuffer size and capacity for diagnostics.
- Add extra fields to
spead2.recv.StreamStats
. - Add support for pcap files to the C++ version of spead2_recv.
- Update the vendored pybind11 to 2.2.4 (fixes some warnings on Python 3.7).
- Deprecate netmap support in documentation.
1.9.2
- autotools are no longer required to install the C++ build (when installing from a release tarball).
1.9.1
- Make
spead2.recv.asyncio.Stream.get()
always yield to the event loop even if there is a heap ready. - Avoid
spead2.recv.asyncio.Stream.get()
holding onto a reference to the heap (via a future) for longer than necessary.
1.9.0
- Add support for TCP/IP (contributed by Rodrigo Tobar).
- Changed command-line options for
spead2_send/spead2_recv:
--ibv
and--netmap
are now boolean flags, and the interface address is set with--bind
. - Added option to specify interface address for
spead2::send::udp_stream
even when not using the multicast constructors. - Constructors that take an existing socket now expect the user to set all
socket options. The old versions that take a socket buffer size are
deprecated. Note that the behaviour of
spead2::send::udp_stream
with a socket has changed: if no buffer size is given, it is left at the OS default, rather than applying the spead2 default. - Fix a bug causing undefined behaviour if a send class is destroyed while there is still data in flight.
Version 1.8.0
- Add In-process transport
- Fix unit testing on Python 3.7
- Add
spead2::send::heap::get_item()
- Support asynchronous iterator protocol for
spead2.recv.asyncio.Stream
(in Python 3.5+).
Version 1.7.2
- Add progress reports to mcdump
- Add ability to pass
-
as filename to mcdump to skip file writing. - Add
--count
option to mcdump
Version 1.7.1
There are no code changes, but this release fixes a packaging error in 1.7.0 that prevented the asyncio integration from being included.
Version 1.7.0
- Support for pcap files. Files passed to spead2_recv.py are now assumed to be pcap files, rather than raw concatenated packets.
- Only log warnings about the ringbuffer being full if at least one stream
reader is lossy (indicated by a new virtual member function in
spead2::recv::Reader
).
Version 1.6.0
- Change spead2_send.py and spead2_send to interpret
the
--rate
option as Gb/s and not Gib/s. - Change send rate limiting to bound the rate at which we catch up if we fall
behind. This is controlled by a new attribute of
StreamConfig
. - Add report at end of spead2_send.py and spead2_send on the actual number of bytes sent and achieved rate.
- Fix a race condition where the stream statistics might only be updated after the stream ended (which lead to unit test failures in some cases).
Version 1.5.2
- Report statistics when spead2_recv.py is stopped by SIGINT.
- Add –ttl option to spead2_send.py and spead2_send.
Version 1.5.1
- Explicitly set UDP checksum to 0 in IBV sender, instead of leaving arbitrary values.
- Improved documentation of asyncio support.
Version 1.5.0
- Support for asyncio in Python 3. For each trollius module there is now an equivalent asyncio module. The installed utilities use asyncio on Python 3.4+.
- Add
spead2.recv.Stream.stop_on_stop_item
to allow a stream to keep receiving after a stop item is received. - Switch shutdown code to use atexit instead of a capsule destructor, to support PyPy.
- Test PyPy support with Travis.
Version 1.4.0
- Remove
--bind
option to spead2_recv.py and spead2_recv. Instead, usehost:port
as the source. This allows subscribing to multiple multicast groups. - Improved access to information about incomplete heaps
(
spead2.recv.IncompleteHeap
type). - Add
MemoryPool.warn_on_empty
control. - Add warning when a stream ringbuffer is full.
- Add statistics to streams.
- Fix spead2_send.py to send a stop heap when using
--heaps
. It was acccidentally broken in 1.2.0. - Add support for packet timestamping in mcdump.
- Return the previous logging function from
spead2::set_log_function()
. - Make Python logging from C++ code asynchronous, to avoid blocking the thread pool on the GIL.
- Upgrade to pybind11 2.2.1 internally.
- Some fixes for PyPy support.
Version 1.3.2
- Fix segfault in shutdown for
spead2_recv.py
(fixes #56). - Fix for
TypeError
in Python 3.6 when reading fields that aren’t aligned to byte boundaries. - Include binary wheels in releases.
Version 1.3.1
- Fix multi-endpoint form of
spead2.recv.Stream.add_udp_ibv_reader()
.
Version 1.3.0
- Rewrite the Python wrapping using pybind11. This should not cause any
compatibility problems, unless you’re using the
spead2/py_*.h
headers. - Allow passing
std::shared_ptr
to constructors that take a thread pool, with the constructed object holding a reference. - Prevent constructing a
spead2.recv.Stream
withmax_heaps=0
(fixes #54).
Version 1.2.2
- Fix rate limiting causing longer sleeps than necessary (fixes #53).
Version 1.2.1
- Disable LTO by default and require the user to opt in, because even if the compiler supports it, linking can still fail (fixes #51).
Version 1.2.0
- Support multiple endpoints for one
udp_ibv_reader
(fixes #48). - Fix compilation on OS X 10.9 (fixes #49)
- Fix
spead2::ringbuffer<T>::emplace()
andspead2::ringbuffer<T>::try_emplace()
- Improved error messages when passing invalid arguments to mcdump
Version 1.1.2
- Only log descriptor replacement if it actually replaces an existing name or ID (regression in 1.1.1).
- Fix build on ARM where compiling against asio requires linking against pthread.
- Updated and expanded performance tuning guide.
Version 1.1.1
- Report the item name in exception for “too few elements for shape” errors
- Overhaul of rules for handling item descriptors that change the name or ID of an item. This prevents stale items from hanging around when the sender changes the name of an item but keeps the same ID, which can cause unrelated errors on the receiver if the shape also changes.
Version 1.1.0
- Allow heap cnt to be set explicitly by sender, and the automatic heap cnt sequence to be specified as a start value and step.
Version 1.0.1
- Fix exceptions to include more information about the source of the failure
- Add mcdump tool
Version 1.0.0
- The C++ API installation has been changed to use autoconf and automake. As a
result, it is possible to run
make install
and get the static library, headers, and tools installed. - The directory structure has changed. The
spead2_*
tools are now installed, example code is now in theexamples
directory, and the headers have moved toinclude/spead2
. - Add support for sending data using libibverbs API (previously only supported for receiving)
- Fix async_send_heap (in Python) to return a future instead of being a coroutine: this fixes a problem with undefined ordering in the trollius example.
- Made sending streams polymorphic, with abstract base class
spead2::send::stream
, to simplify writing generic code that can operate on any type of stream. This will break code that depended on the old template class of the same name, which has been renamed tospead2::send::stream_impl
. - Add
--memcpy-nt
to spead2_recv.py and spead2_bench.py - Multicast support in spead2_bench.py and spead2_bench
- Changes to the algorithm for spead2_bench.py and spead2_bench: it now starts by computing the maximum send speed, and then either reporting that this is the limiting factor, or using it to start the binary search for the receive speed. It is also stricter about lost heaps.
- Some internal refactoring of code for dealing with raw packets, so that it is shared between the netmap and ibv readers.
- Report function name that failed in semaphore system_error exceptions.
- Make the unit tests pass on OS X (now tested on travis-ci.org)
Version 0.10.4
- Refactor some of the Boost.Python glue code to make it possible to reuse parts of it in writing new Python extensions that use the C++ spead2 API.
Version 0.10.3
- Suppress “operation aborted” warnings from UDP reader when using the API to stop a stream (introduced in 0.10.0).
- Improved elimination of duplicate item pointers, removing them as they’re received rather than when freezing a live heap (fixes #46).
- Use hex for reporting item IDs in log messages
- Fix reading from closed file descriptor after stream.stop() (fixes #42)
- Fix segmentation fault when using ibverbs but trying to bind to a non-RDMA device network interface (fixes #45)
Version 0.10.2
- Fix a performance problem when a heap contains many packets and every packet contains item pointers. The performance was quadratic instead of linear.
Version 0.10.1
- Fixed a bug in registering add_udp_ibv_reader in Python, which broke spead2_recv.py, and possibly any other code using this API.
- Fixed spead2_recv.py ignoring
--ibv-max-poll
option
Version 0.10.0
- Added support for libibverbs for improved performance in both Python and C++.
- Avoid per-packet shared_ptr reference counting, accidentally introduced in 0.9.0, which caused a small performance regression. This is unfortunately a breaking change to the interface for implementing custom memory allocators.
Version 0.9.1
- Fix using a
MemoryPool
with a thread pool and low water mark (regression in 0.9.0).
Version 0.9.0
- Add support for custom memory allocators.
Version 0.8.2
- Ensure correct operation when loop=None is passed explicitly to trollius stream constructors, for consistency with functions that have it as a keyword parameter.
Version 0.8.1
- Suppress
recvmmsg: resource temporarily unavailable
warnings (fixes #43)
Version 0.8.0
- Extend
MemoryPool
to allow a background thread to replenish the pool when it gets low. - Extend
ThreadPool
to allow the user to pin the threads to specific CPU cores (on glibc).
Version 0.7.1
- Fix ring_stream destructor to not deadlock (fixes #41)
Version 0.7.0
- Change handling of incomplete heaps (fixes #39). Previously, incomplete heaps were only abandoned once there were more than max_heaps of them. Now, they are abandoned once max_heaps more heaps are seen, even if those heaps were complete. This causes the warnings for incomplete heaps to appear closer to the time they arrived, and also has some extremely small performance advantages due to changes in the implementation.
- backwards-incompatible change: remove
set_max_heaps()
. It was not previously documented, so hopefully is not being used. It could not be efficiently supported with the design changes above. - Add
spead2.recv.Stream.set_memcpy()
to control non-temporal caching hints. - Fix C++ version of spead2_bench to actually use the memory pool
- Reduce memory usage in spead2_bench (C++ version)
Version 0.6.3
- Partially fix #40:
set_max_heaps()
andset_memory_pool()
will no longer deadlock if called on a stream that has already had a reader added and is receiving data.
Version 0.6.2
- Add a fast path for integer items that exactly fit in an immediate.
- Optimise Python code by replacing np.product with a pure Python implementation.
Version 0.6.1
- Filter out duplicate items from a heap. It is undefined which of a set of
duplicates will be retained (it was already undefined for
spead2.ItemGroup
).
Version 0.6.0
- Changed item versioning on receive to increment version number on each update rather that setting to heap id. This is more robust to using a single item or item group with multiple streams, and most closely matches the send path.
- Made the protocol enums from the C++ library available in the Python library as well.
- Added functions to create stream start items (send) and detect them (recv).
Version 0.5.0
- Added friendlier support for multicast. When a multicast address is passed
to
add_udp_reader()
, the socket will automatically join the multicast group and setSO_REUSEADDR
so that multiple sockets can consume from the same stream. There are also new constructors and methods to give explicit control over the TTL (send) and interface (send and receive), including support for IPv6.
Version 0.4.7
- Added in-memory mode to the C++ version of spead2_bench, to measure the packet handling speed independently of the lossy networking code
- Optimization to duplicate packet checks. This makes a substantial performance improvement when using small (e.g. 512 byte) packets and large heaps.
Version 0.4.6
- Fix a data corruption (use-after-free) bug on send side when data is being sent faster than the socket can handle it.
Version 0.4.5
- Fix bug causing some log messages to be remapped to DEBUG level
Version 0.4.4
- Increase log level for packet rejection from DEBUG to INFO
- Some minor optimisations
Version 0.4.3
- Handle heaps that have out-of-range item offsets without crashing (#32)
- Fix handling of heaps without heap length headers
spead2.send.UdpStream.send_heap()
now correctly raisesIOError
if the heap is rejected due to being full, or if there was an OS-level error in sending the heap.- Fix
spead2.send.trollius.UdpStream.async_send_heap()
for the case where the last sent heap failed. - Use eventfd(2) for semaphores on Linux, which makes a very small improvement in ringbuffer performance.
- Prevent messages about descriptor replacements for descriptor reissues with no change.
- Fix a use-after-free bug (affecting Python only).
- Throw
OverflowError
on out-of-range UDP port number, instead of wrapping.
Version 0.4.2
- Fix compilation on systems without glibc
- Fix test suite for non-Linux systems
- Add
spead2.send.trollius.UdpStream.async_flush()
Version 0.4.1
- Add C++ version of spead2_recv, a more fully-featured alternative to test_recv
- backwards-incompatible change:
Add ring_heaps parameter to
ring_stream
constructor. Code that specifies the contiguous_only parameter will need to be modified since the position has changed. Python code is unaffected. - Increased the default for ring_heaps from 2 (previously hardcoded) to 4 to improve throughput for small heaps.
- Add support for user to provide the socket for UDP communications. This allows socket options to be set by the user, for example, to configure multicast.
- Force numpy>=1.9.2 to avoid a numpy [bug](https://github.com/numpy/numpy/issues/5356).
- Add experimental support for receiving packets via netmap
- Improved receive performance on Linux, particularly for small packets, using [recvmmsg](http://linux.die.net/man/2/recvmmsg).
Version 0.4.0
- Enforce ASCII encoding on descriptor fields.
- Warn if a heap is dropped due to being incomplete.
- Add –ring option to C++ spead2_bench to test ringbuffer performance.
- Reading from a memory buffer (e.g. with
add_buffer_reader()
) is now reliable, instead of dropping heaps if the consumer doesn’t keep up (heaps can still be dropped if packets extracted from the buffer are out-of-order, but it is deterministic). - The receive ringbuffer now has a fixed size (2), and pushes are blocking. The result is lower memory usage, and it is no longer necessary to pass a large max_heaps value to deal with the consumer not always keeping up. Instead, it may be necessary to increase the socket buffer size.
- backwards-incompatible change:
Calling
spead2::recv::ring_stream::stop()
now discards remaining partial heaps instead of adding them to the ringbuffer. This only affects the C++ API, because the Python API does not provide any access to partial heaps anyway. - backwards-incompatible change:
A heap with a stop flag is swallowed rather than passed to
heap_ready()
(see issue [#29](https://github.com/ska-sa/spead2/issues/29)).
Version 0.3.0
This release contains a number of backwards-incompatible changes in the Python bindings, although most uses will probably not notice:
- When a received character array is returned as a string, it is now of type
str
(previously it wasunicode
in Python 2). - An array of characters with a numpy descriptor with type S1 will no longer automatically be turned back into a string. Only using a format of [(‘c’, 8)] will do so.
- The c format code may now only be used with a length of 8.
- When sending, values will now always be converted to a numpy array first, even if this isn’t the final representation that will be put on the network. This may lead to some subtle changes in behaviour.
- The BUG_COMPAT_NO_SCALAR_NUMPY introduced in 0.2.2 has been removed. Now, specifying an old-style format will always use that format at the protocol level, rather than replacing it with a numpy descriptor.
There are also some other bug-fixes and improvements:
- Fix incorrect warnings about send buffer size.
- Added –descriptors option to spead2_recv.py.
- The dtype argument to
spead2.ItemGroup.add_item()
is now optional, removing the need to specify dtype=None when passing a format.
Version 0.2.2
- Workaround for a PySPEAD bug that would cause PySPEAD to fail if sent a simple scalar value. The user must still specify scalars with a format rather than a dtype to make things work.
Version 0.2.1
- Fix compilation on OS X again. The extension binary will be slightly larger as a result, but still much smaller than before 0.2.0.
Version 0.2.0
- backwards-incompatible change: for sending, the heap count is now tracked
internally by the stream, rather than an attribute of the heap. This affects
both C++ and Python bindings, although Python code that always uses
HeapGenerator
rather than directly creating heaps will not be affected. - The
HeapGenerator
is extended to allow items to be added to an existing heap and to give finer control over whether descriptors and/or values are put in the heap. - Fixes a bug that caused some values to be cast to non-native endian.
- Added overloaded equality tests on Flavour objects.
- Strip the extension binary to massively reduce its size
Version 0.1.2
- Coerce values to int for legacy ‘u’ and ‘i’ fields
- Fix flavour selection in example code
Version 0.1.1
- Fixes to support OS X
Version 0.1.0
- First public release