[collectd] ESTP protocol and collectd plugin

[Paul Colomiets]

Hi,

Few weeks ago I've started to discuss a standard protocol for
statistics submission to use with Zeromq and Crossroads IO. Here is
what we came up with at the moment:

https://github.com/estp/estp/blob/master/specification.rst

I'm here for two main goals:

1. Discuss the protocol with statistics gurus

2. Propose to add the plugin which implements ESTP over Zeromq into
collectd (and the one for crossroads too, in the future)


The protocol is text-based and in designed to be human-readable and is
parsed super-easily. The questions to the collectd community are
following:

1. The DERIVE data source type is used as a COUNTER (with minimum of
0) in every type, even with type called "derive". So are there no use
cases for real DERIVE type?

2. The ABSOLUTE data source type is not used in the plugins and is
discouraged in docs. With ESTP I'm trying to encourage using ABSOLUTE
type (called "delta" in protocol), as it can be easier to be processed
by the small stateless application (which is one of the major use
cases ESTP is modelled for). Are there any issues of using ABSOLUTE
instead of COUNTER in application which can subtract two counter
values? Or is there a reason to store ABSOLUTE as GAUGE (by dividing
it by interval if needed) ?


The plugin code is here:

https://github.com/estp/collectd-zeromq-estp

To achieve the goals of the protocol, the mapping the collectd data
types into ESTP is not lossless. But there is an extension that allows
to keep lossless type info, still allowing other clients that doesn't
know about collectd extension to parse basic data. So the plugin can
be used as a network transport between several collectd instances.
Here is the spec:

http://github.com/estp/estp/blob/master/collectd-extension.rst

What are the requirements for the plugin to be included into collectd?

Also what's recommended way for building the plugin that is not in
collectd source? I've written simple makefile that uses full collectd
source that's checked out as submodule. Is there a better way?

Any comments are appreciated.

-- 
Paul