Support for ibverbs

The support for libibverbs is essentially the same as for Python, with the same limitations. The programmatic interface is via the spead2::recv::udp_ibv_reader and spead2::send::udp_ibv_stream classes:

class spead2::recv::udp_ibv_config : public spead2::detail::udp_ibv_config_base<udp_ibv_config>

Configuration for udp_ibv_reader.

Subclassed by spead2::recv::udp_ibv_config_wrapper

Public Functions

std::size_t get_max_size() const

Get maximum packet size to accept.

udp_ibv_config &set_max_size(std::size_t max_size)

Set maximum packet size to accept.

const std::vector<boost::asio::ip::udp::endpoint> &get_endpoints() const

Get the configured endpoints.

udp_ibv_config &set_endpoints(const std::vector<boost::asio::ip::udp::endpoint> &endpoints)

Set the endpoints (replacing any previous).

Exceptions

  • std::invalid_argument: if any element of endpoints is invalid.

udp_ibv_config &add_endpoint(const boost::asio::ip::udp::endpoint &endpoint)

Append a single endpoint.

Exceptions

  • std::invalid_argument: if endpoint is invalid.

const boost::asio::ip::address get_interface_address() const

Get the currently set interface address.

udp_ibv_config &set_interface_address(const boost::asio::ip::address &interface_address)

Set the interface address.

Exceptions

  • std::invalid_argument: if interface_address is not an IPv4 address.

std::size_t get_buffer_size() const

Get the currently configured buffer size.

udp_ibv_config &set_buffer_size(std::size_t buffer_size)

Set the buffer size.

The value 0 is special and resets it to the default. The actual buffer size used may be slightly different to round it to a whole number of packet-sized slots.

int get_comp_vector() const

Get the completion channel vector (see set_comp_vector)

udp_ibv_config &set_comp_vector(int comp_vector)

Set the completion channel vector (interrupt) for asynchronous operation.

Use a negative value to poll continuously. Polling should not be used if there are other users of the thread pool. If a non-negative value is provided, it is taken modulo the number of available completion vectors. This allows a number of streams to be assigned sequential completion vectors and have them load-balanced, without concern for the number available.

int get_max_poll() const

Get maximum number of times to poll in a row (see set_max_poll)

udp_ibv_config &set_max_poll(int max_poll)

Set maximum number of times to poll in a row.

If interrupts are enabled (default), it is the maximum number of times to poll before waiting for an interrupt; if they are disabled by set_comp_vector, it is the number of times to poll before letting other code run on the thread.

Exceptions

  • std::invalid_argument: if max_poll is zero.

Public Static Attributes

constexpr std::size_t default_buffer_size = 16 * 1024 * 1024

Receive buffer size, if none is explicitly set.

constexpr std::size_t default_max_size = udp_reader_base::default_max_size

Maximum packet size to accept, if none is explicitly set.

constexpr int default_max_poll = 10

Number of times to poll in a row, if none is explicitly set.

class spead2::recv::udp_ibv_reader : public spead2::recv::detail::udp_ibv_reader_base<udp_ibv_reader>

Synchronous or asynchronous stream reader that reads UDP packets using the Infiniband verbs API.

It currently only supports IPv4, with no fragmentation, IP header options, or VLAN tags.

Public Functions

udp_ibv_reader(stream &owner, const boost::asio::ip::udp::endpoint &endpoint, const boost::asio::ip::address &interface_address, std::size_t max_size = udp_ibv_config::default_max_size, std::size_t buffer_size = udp_ibv_config::default_buffer_size, int comp_vector = 0, int max_poll = udp_ibv_config::default_max_poll)

Constructor with single endpoint (deprecated).

Parameters

  • owner: Owning stream

  • endpoint: Address and port. Note that is it possible for the address to be unicast and different to the interface_address: the interface may have multiple IP addresses, in which case this filters packets on the interface by IP address. An unspecified address can also be used to skip address filtering.

  • interface_address: Address of the interface which should join the group and listen for data

  • max_size: Maximum packet size that will be accepted

  • buffer_size: Requested memory allocation for work requests. Note that this is used to determine the number of packets to buffer; if the packets are smaller than max_size, then fewer bytes will be buffered.

  • comp_vector: Completion channel vector (interrupt) for asynchronous operation, or a negative value to poll continuously. Polling should not be used if there are other users of the thread pool. If a non-negative value is provided, it is taken modulo the number of available completion vectors. This allows a number of readers to be assigned sequential completion vectors and have them load-balanced, without concern for the number available.

  • max_poll: Maximum number of times to poll in a row, without waiting for an interrupt (if comp_vector is non-negative) or letting other code run on the thread (if comp_vector is negative).

Exceptions

  • std::invalid_argument: If endpoint is specified and is not an IPv4 address

  • std::invalid_argument: If interface_address is not an IPv4 address

udp_ibv_reader(stream &owner, const std::vector<boost::asio::ip::udp::endpoint> &endpoints, const boost::asio::ip::address &interface_address, std::size_t max_size = udp_ibv_config::default_max_size, std::size_t buffer_size = udp_ibv_config::default_buffer_size, int comp_vector = 0, int max_poll = udp_ibv_config::default_max_poll)

Constructor with multiple endpoints (deprecated).

Parameters

  • owner: Owning stream

  • endpoints: Addresses and ports. Note that is it possible for the addresses to be unicast and different to the interface_address: the interface may have multiple IP addresses, in which case this filters packets on the interface by IP address. An unspecified address can also be used to skip address filtering.

  • interface_address: Address of the interface which should join the group and listen for data

  • max_size: Maximum packet size that will be accepted

  • buffer_size: Requested memory allocation for work requests. Note that this is used to determine the number of packets to buffer; if the packets are smaller than max_size, then fewer bytes will be buffered.

  • comp_vector: Completion channel vector (interrupt) for asynchronous operation, or a negative value to poll continuously. Polling should not be used if there are other users of the thread pool. If a non-negative value is provided, it is taken modulo the number of available completion vectors. This allows a number of readers to be assigned sequential completion vectors and have them load-balanced, without concern for the number available.

  • max_poll: Maximum number of times to poll in a row, without waiting for an interrupt (if comp_vector is non-negative) or letting other code run on the thread (if comp_vector is negative).

Exceptions

  • std::invalid_argument: If endpoints is empty.

  • std::invalid_argument: If any element of endpoints is specified and is not an IPv4 address

  • std::invalid_argument: If interface_address is not an IPv4 address

udp_ibv_reader(stream &owner, const udp_ibv_config &config)

Constructor.

Parameters

  • owner: Owning stream

  • config: Configuration

Exceptions

  • std::invalid_argument: If no endpoints are set.

  • std::invalid_argument: If no interface address is set.

class spead2::send::udp_ibv_config : public spead2::detail::udp_ibv_config_base<udp_ibv_config>

Configuration for udp_ibv_stream.

Subclassed by spead2::send::udp_ibv_config_wrapper

Public Functions

std::uint8_t get_ttl() const

Get the IP TTL.

udp_ibv_config &set_ttl(std::uint8_t ttl)

Set the IP TTL.

const std::vector<memory_region> &get_memory_regions() const

Get currently registered memory regions.

udp_ibv_config &set_memory_regions(const std::vector<memory_region> &memory_regions)

Register a set of memory regions (replacing any previous).

Items stored inside such pre-registered memory regions can (in most cases) be transmitted without making a copy. A memory region is defined by a start pointer and a size in bytes.

Memory regions must not overlap; this is only validating when constructing the stream.

udp_ibv_config &add_memory_region(const void *ptr, std::size_t size)

Append a memory region (see set_memory_regions)

const std::vector<boost::asio::ip::udp::endpoint> &get_endpoints() const

Get the configured endpoints.

udp_ibv_config &set_endpoints(const std::vector<boost::asio::ip::udp::endpoint> &endpoints)

Set the endpoints (replacing any previous).

Exceptions

  • std::invalid_argument: if any element of endpoints is invalid.

udp_ibv_config &add_endpoint(const boost::asio::ip::udp::endpoint &endpoint)

Append a single endpoint.

Exceptions

  • std::invalid_argument: if endpoint is invalid.

const boost::asio::ip::address get_interface_address() const

Get the currently set interface address.

udp_ibv_config &set_interface_address(const boost::asio::ip::address &interface_address)

Set the interface address.

Exceptions

  • std::invalid_argument: if interface_address is not an IPv4 address.

std::size_t get_buffer_size() const

Get the currently configured buffer size.

udp_ibv_config &set_buffer_size(std::size_t buffer_size)

Set the buffer size.

The value 0 is special and resets it to the default. The actual buffer size used may be slightly different to round it to a whole number of packet-sized slots.

int get_comp_vector() const

Get the completion channel vector (see set_comp_vector)

udp_ibv_config &set_comp_vector(int comp_vector)

Set the completion channel vector (interrupt) for asynchronous operation.

Use a negative value to poll continuously. Polling should not be used if there are other users of the thread pool. If a non-negative value is provided, it is taken modulo the number of available completion vectors. This allows a number of streams to be assigned sequential completion vectors and have them load-balanced, without concern for the number available.

int get_max_poll() const

Get maximum number of times to poll in a row (see set_max_poll)

udp_ibv_config &set_max_poll(int max_poll)

Set maximum number of times to poll in a row.

If interrupts are enabled (default), it is the maximum number of times to poll before waiting for an interrupt; if they are disabled by set_comp_vector, it is the number of times to poll before letting other code run on the thread.

Exceptions

  • std::invalid_argument: if max_poll is zero.

Public Static Attributes

constexpr std::size_t default_buffer_size = 512 * 1024

Default send buffer size.

constexpr int default_max_poll = 10

Default number of times to poll in a row.

class spead2::send::udp_ibv_stream : public spead2::send::stream

Public Functions

udp_ibv_stream(io_service_ref io_service, const boost::asio::ip::udp::endpoint &endpoint, const stream_config &config, const boost::asio::ip::address &interface_address, std::size_t buffer_size = udp_ibv_config::default_buffer_size, int ttl = 1, int comp_vector = 0, int max_poll = udp_ibv_config::default_max_poll)

Backwards-compatibility constructor (taking only a single endpoint).

Refer to udp_ibv_config for an explanation of the arguments.

Exceptions

  • std::invalid_argument: if endpoint is not an IPv4 multicast address

  • std::invalid_argument: if interface_address is not an IPv4 address

udp_ibv_stream(io_service_ref io_service, const stream_config &config, const udp_ibv_config &ibv_config)

Constructor.

Parameters

  • io_service: I/O service for sending data

  • config: Common stream configuration

  • ibv_config: Class-specific stream configuration

Exceptions

  • std::invalid_argument: if ibv_config does not have an interface address set.

  • std::invalid_argument: if ibv_config does not have any endpoints set.

  • std::invalid_argument: if memory regions overlap.