The binary protocol is the protocol implemented by the Network plugin and some external implementations to exchange data collected by collectd or to send data to an instance of collectd.
Until the Network plugin has been factorized out into a library, it is useful to have some documentation to reimplement it.
- Default UDP port
- Default IPv4 Multicast group
- Default IPv6 multicast group
- Maximum packet size
- 1024 bytes (payload only, not including UDP / IP headers)
In versions 4.0 through 4.7, the receive buffer has a fixed size of 1024 bytes. When longer packets are received, the trailing data is simply ignored. Since version 4.8, the buffer size can be configured. Version 5.0 will increase the default buffer size to 1452 bytes (the maximum payload size when using UDP/IPv6 over Ethernet).
Each packet consists of one or more so called “parts”. Each part starts with the same four bytes: Two bytes that specify the “part type” (what kind of information is enclosed in the part) and two bytes which specify the length of the part, including the four header bytes itself. Both type and length should be encoded in network order, aka Big Endian. The maximum length of payload in any part is therefore 65531 bytes.
Using this layout, clients can determine the length of a part they don't know and lets them skip unknown data. This makes the protocol forward compatible so that new features can be added easily.
There are two part layouts that are used for a couple of “types”: numeric (an 8 byte integer) and string.
Numeric integer values, e. g. the interval and time values, are transferred using 8 byte integers. The length field of those parts must therefore always be set to 12.
Strings are transferred including a null byte at the end. In the example you can see the encoding of the string “foobar”. The string is six characters long, followed by a null-byte and appended to a four byte header, leading to a length of 11 bytes for this part.
Value parts encode an actual data sample. Preceding time, plugin, plugin instance, type, and type instance parts must have set the context for the data sample. The value part consists of:
- Length (16 bit field)
- Number of values (16 bit field)
- Values, for each one:
- Data type code (8 bit field)
- Value (64 bit field)
COUNTER→ network (big endian) unsigned integer
GAUGE→ x86 (little endian) double
DERIVE→ network (big endian) signed integer
ABSOLUTE→ network (big endian) unsigned integer
- Data type code (8 bit field)
Many data samples have a single value, such as "cpu" (see types.db(5)), but others have multiple values, such as "disk_merged", which has read and write values.
In case you have many values, the types need to be defined first, then the values, like this: [type][type][type][value][value][value] and not [type][value][type][value][type][value].
The following numeric types are currently used to identify the type of a “part”. Defines are available from src/network.h.
||Host||String||The name of the host to associate with subsequent data values|
||Time||Numeric||The timestamp to associate with subsequent data values, unix time format (seconds since epoch)|
||Time (high resolution)||Numeric||The timestamp to associate with subsequent data values. Time is defined in 2–30 seconds since epoch. New in Version 5.0.|
||Plugin||String||The plugin name to associate with subsequent data values, e.g. "cpu"|
||Plugin instance||String||The plugin instance name to associate with subsequent data values, e.g. "1"|
||Type||String||The type name to associate with subsequent data values, e.g. "cpu"|
||Type instance||String||The type instance name to associate with subsequent data values, e.g. "idle"|
||Values||other||Data values, see above|
||Interval||Numeric||Interval used to set the "step" when creating new RRDs unless rrdtool plugin forces StepSize. Also used to detect values that have timed out.|
||Interval (high resolution)||Numeric||The interval in which subsequent data values are collected. The interval is given in 2–30 seconds. New in Version 5.0.|
||Signature (HMAC-SHA-256)||other (todo)|
||Encryption (AES-256/OFB/SHA-1)||other (todo)|
- Small Python script by Adrian Perez
- collectd Python module by Eli Courtwright.
- PacketWriter.java of jcollectd
- Of course the Network plugin (src/network.c) of collectd itself
- Ruby implementation by Astro.
- erlang-collectd, an Erlang implementation by Astro