Cool, I'll clean up the code and put the changes up on github.<div><br></div><div>As it stands, my plugin only receives data -- fully implementing all the features of the unixsock version in libevent will take a bit more work, but I should be able to reuse all the parsing/dispatching code from unixsock. (That'll make using a buffer-chain i/o model more difficult, but that's probably unnecessary if most of the messages from the clients are small)<br>
<br></div><div>For the current implementation, I've written a couple of python clients: one ordinary, synchronous python module and one Twisted client. </div><div><br></div><div>I'm going to be out for most of the weekend, but I'll see about making everything available on Monday.</div>
<div><br></div><div>Also, I've already started integrating this in to some of our Twisted daemons here at Meebo, with great results. Being able to collect trending data from a daemon with one line of python and no configuration changes anywhere is pretty fantastic. I'm pretty stoked about how well it turned out. :)</div>
<div><br></div><div>--Shaun</div><div><br><div class="gmail_quote">On Sat, Jun 12, 2010 at 1:56 AM, Florian Forster <span dir="ltr"><<a href="mailto:octo@verplant.org">octo@verplant.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi Shaun,<br>
<div class="im"><br>
On Fri, Jun 04, 2010 at 12:39:47PM -0700, Shaun Lindsay wrote:<br>
> So, I wrote a plugin I'm calling aggregator to do handle this sort of<br>
> use case.<br>
<br>
</div>what you're describing sounds a lot like the "Event infrastructure", an<br>
idea that has been floating around for a long time by nobody actually<br>
implemented it yet.<br>
<br>
The idea was to implement a new command in the "UnixSock" plugin with<br>
the following syntax:<br>
<br>
EVENT <Name> [<Number>]<br>
<br>
When this command is received, the "UnixSock" plugin looks up the<br>
structure called <Name> or allocates it if necessary. If a <Number> is<br>
given, the number is added to a counter in the structure, otherwise the<br>
counter is increased by one.<br>
<br>
So, for example, if a web-application issues<br>
<br>
EVENT "pageview"<br>
<br>
after each page served, you'll end up with a graph showing the pageviews<br>
per second.<br>
<br>
A possible extension would be to allow the user to specify a "data<br>
set" / "type" to use. The same web-application could, for example, issue<br>
this command<br>
<br>
EVENT type="total_bytes" "pages" 18063<br>
<br>
to use the "total_bytes" data set and add 18 kByte to the current<br>
counter.<br>
<div class="im"><br>
> Again due to my particular use case, it needed to be able to handle a<br>
> large number of concurrent connections (50k potentially), so rather<br>
> than spawning a thread per connection ala unixsock, I kick off one<br>
> thread when the plugin init's and then run a libevent server inside<br>
> that thread and do everything asynchronously.<br>
<br>
</div>I think it might be possible to incorporate that into the "UnixSock"<br>
plugin, either as a compile-time or run-time choice. We could, for<br>
example, use libevent if it is available and fall back to the current<br>
implementation otherwise.<br>
<div class="im"><br>
> So, then, two questions: First, did I just reinvent the wheel on<br>
> this? Is there a plugin that already satisfies this sort of need?<br>
<br>
</div>No. The idea is not new, but there is not code in the daemon / plugins<br>
yet.<br>
<div class="im"><br>
> Second, if this is new, is anyone else interested in this sort of<br>
> functionality?<br>
<br>
</div>Definitely ;)<br>
<br>
Regards,<br>
—octo<br>
<br>
[0] <<a href="http://collectd.org/wiki/index.php/Roadmap#Event_infrastructure" target="_blank">http://collectd.org/wiki/index.php/Roadmap#Event_infrastructure</a>><br>
<font color="#888888">--<br>
Florian octo Forster<br>
Hacker in training<br>
GnuPG: 0x91523C3D<br>
<a href="http://verplant.org/" target="_blank">http://verplant.org/</a><br>
</font><br>-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v1.4.6 (GNU/Linux)<br>
<br>
iD8DBQFME0uoHdggu3Q05IYRApv6AJ9ha9VfFl0M/TUtXQyeewhgbimOBgCfU/aa<br>
N6PfB+Fh3x4SkhumjToh+RU=<br>
=nxAo<br>
-----END PGP SIGNATURE-----<br>
<br></blockquote></div><br></div>