[collectd] Dynamic Intervals and Pulling Data

[Florian Forster]

Hey Paul :)

On Mon, Sep 07, 2009 at 06:54:19PM -0700, Paul Querna wrote:
> I am looking at a way to contribute a plugin for collectd to allow
> 'live' streaming of data.
> 
> Of course, setting the interval to 1 second, and using the existing
> network plugin, would basically achieve this, but for 99.9% of the
> time, running all of the plugins every second would be massive
> overkill.

I understand the goal is to provide a live streaming feature to users.
When noone is watching, updating the value every few seconds is
perfectly fine, but when the user is wathing the vallue “live” the
value should be collected much more frequently.

Currently such stuff is done using the “unixsock” plugin [0]. The show-
stopper is currently that it's not opening an IPv6/IPv4 socket but a
UNIX domain socket.

I started writing a new plugin called “netcmd” which works just like the
“unixsock” plugin but can use IPv6, IPv4 and UNIX domain sockets. Some
initial (unfinished, probably broken) code is available in the
“ff/netcmd” branch of the Git repository. A few people showed general
interest in the plugin, but so far nobody stepped up to polish it up for
inclusion …

Once that's available I guess the best option is to implement a
“SETINTERVAL” command to change the interval of a callback on-demand.

(Implementing this command is probably a bit tricky due to the way
read-callbacks are scheduled, though. Also, the names of the callbacks
might be a bit tricky to find out. A “LISTCALLBACK” command might be
necessary. Maybe it's best to discuss those details in IRC …)

> I think the best design I've thought of so far is to creare a new
> plugin, tentatively called network-tcp.  It would connect to an
> IP/port on startup, stream data in the same format as the UDP plugin
> to the end point.

The suggestion of a TCP based network plugin has been discussed before,
especially to detect when the remote side is gone and cache updates
until it is back again. I thought that maybe this could be implemented
into the “netcmd” plugin as well, possibly as “STREAM START” /
“STREAM STOP” commands.

> I've looked at a few of the existing plugins, most of them don't use
> the interval_g, but a few do, mostly in calculating the period over
> which they were ran for averages.

Basically all plugins use interval_g to set value_list_t.interval. It's
obscured by the “VALUE_LIST_INIT” macro, though. This value is then used
to calculate rates, configure the RRDtool plugin and so on. I think the
currently active interval would have to be made available to the plugins
somehow so they can set this correctly.

A bit hack'ish but easier to implement would be to make this information
available using pthread_setspecific(3) and set the interval member in
plugin_dispatch_values.

Regards,
-octo

[0] <http://collectd.org/wiki/index.php/Plugin:UnixSock>
-- 
Florian octo Forster
Hacker in training
GnuPG: 0x91523C3D
http://verplant.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mailman.verplant.org/pipermail/collectd/attachments/20090908/7ad73c6e/attachment.pgp