[collectd] unixsock-like plugin extension

[Shaun Lindsay]

Hello all,

I've been looking at collectd for some application trending purposes.  At
first, I thought the unixsock plugin would perfect.  The text-based protocol
is straightforward for pushing data out of an application, which is nice.
 Additionally, it would be easy to write a callback-driven client library to
push that data asynchronously, which is something of a requirement for the
applications I want to push data from.

The one downside is that it depends on whatever is pushing data to collectd
to respect the sampling interval and push data roughly on that timeframe.
 In my case, I'll potentially have thousands of processes per server pushing
data points at arbitrary times (things such as error counts or elapsed times
for certain operations, etc).  I could have another process parse logs and
then push that info to collectd, but that seems a bit silly.

So, I wrote a plugin I'm calling aggregator to do handle this sort of use
case.  It listens on a unix socket and uses a protocol similar to the
unixsock plugin, except that it only accepts data points (rather than
providing access to the full API), and doesn't require timestamps.  Whenever
it sees a new host/plugin/type combination it adds it to a hash table and
starts collecting the data over whatever time interval is specified in the
config.  When that time interval is up, pushes out one value (either the sum
of the submitted values or their average, depending on what's needed).

Again due to my particular use case, it needed to be able to handle a large
number of concurrent connections (50k potentially), so rather than spawning
a thread per connection ala unixsock, I kick off one thread when the plugin
init's and then run a libevent server inside that thread and do everything
asynchronously.

So, then, two questions:  First, did I just reinvent the wheel on this?  Is
there a plugin that already satisfies this sort of need?  Second, if this is
new, is anyone else interested in this sort of functionality?  I'd be glad
to hand off the code.  The only tricky part is that it depends on libevent
and, to a lesser extent, glib-2.0, which makes it a little less clean to
build than the core functionality.

Thanks,
Shaun Lindsay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.verplant.org/pipermail/collectd/attachments/20100604/3caf0495/attachment.htm