[collectd] unixsock-like plugin extension

[Shaun Lindsay]

Cool, I'll clean up the code and put the changes up on github.

As it stands, my plugin only receives data -- fully implementing all the
features of the unixsock version in libevent will take a bit more work, but
I should be able to reuse all the parsing/dispatching code from unixsock.
 (That'll make using a buffer-chain i/o model more difficult, but that's
probably unnecessary if most of the messages from the clients are small)

For the current implementation, I've written a couple of python clients:
 one ordinary, synchronous python module and one Twisted client.

I'm going to be out for most of the weekend, but I'll see about making
everything available on Monday.

Also, I've already started integrating this in to some of our Twisted
daemons here at Meebo, with great results.  Being able to collect trending
data from a daemon with one line of python and no configuration changes
anywhere is pretty fantastic.  I'm pretty stoked about how well it turned
out. :)

--Shaun

On Sat, Jun 12, 2010 at 1:56 AM, Florian Forster <octo at verplant.org> wrote:

> Hi Shaun,
>
> On Fri, Jun 04, 2010 at 12:39:47PM -0700, Shaun Lindsay wrote:
> > So, I wrote a plugin I'm calling aggregator to do handle this sort of
> > use case.
>
> what you're describing sounds a lot like the "Event infrastructure", an
> idea that has been floating around for a long time by nobody actually
> implemented it yet.
>
> The idea was to implement a new command in the "UnixSock" plugin with
> the following syntax:
>
>  EVENT <Name> [<Number>]
>
> When this command is received, the "UnixSock" plugin looks up the
> structure called <Name> or allocates it if necessary. If a <Number> is
> given, the number is added to a counter in the structure, otherwise the
> counter is increased by one.
>
> So, for example, if a web-application issues
>
>  EVENT "pageview"
>
> after each page served, you'll end up with a graph showing the pageviews
> per second.
>
> A possible extension would be to allow the user to specify a "data
> set" / "type" to use. The same web-application could, for example, issue
> this command
>
>  EVENT type="total_bytes" "pages" 18063
>
> to use the "total_bytes" data set and add 18 kByte to the current
> counter.
>
> > Again due to my particular use case, it needed to be able to handle a
> > large number of concurrent connections (50k potentially), so rather
> > than spawning a thread per connection ala unixsock, I kick off one
> > thread when the plugin init's and then run a libevent server inside
> > that thread and do everything asynchronously.
>
> I think it might be possible to incorporate that into the "UnixSock"
> plugin, either as a compile-time or run-time choice. We could, for
> example, use libevent if it is available and fall back to the current
> implementation otherwise.
>
> > So, then, two questions:  First, did I just reinvent the wheel on
> > this?  Is there a plugin that already satisfies this sort of need?
>
> No. The idea is not new, but there is not code in the daemon / plugins
> yet.
>
> > Second, if this is new, is anyone else interested in this sort of
> > functionality?
>
> Definitely ;)
>
> Regards,
> —octo
>
> [0] <http://collectd.org/wiki/index.php/Roadmap#Event_infrastructure>
> --
> Florian octo Forster
> Hacker in training
> GnuPG: 0x91523C3D
> http://verplant.org/
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFME0uoHdggu3Q05IYRApv6AJ9ha9VfFl0M/TUtXQyeewhgbimOBgCfU/aa
> N6PfB+Fh3x4SkhumjToh+RU=
> =nxAo
> -----END PGP SIGNATURE-----
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.verplant.org/pipermail/collectd/attachments/20100612/05248a21/attachment.htm