Hello all,<div><br></div><div>I've been looking at collectd for some application trending purposes. At first, I thought the unixsock plugin would perfect. The text-based protocol is straightforward for pushing data out of an application, which is nice. Additionally, it would be easy to write a callback-driven client library to push that data asynchronously, which is something of a requirement for the applications I want to push data from.</div>
<div><br></div><div>The one downside is that it depends on whatever is pushing data to collectd to respect the sampling interval and push data roughly on that timeframe. In my case, I'll potentially have thousands of processes per server pushing data points at arbitrary times (things such as error counts or elapsed times for certain operations, etc). I could have another process parse logs and then push that info to collectd, but that seems a bit silly.</div>
<div><br></div><div>So, I wrote a plugin I'm calling aggregator to do handle this sort of use case. It listens on a unix socket and uses a protocol similar to the unixsock plugin, except that it only accepts data points (rather than providing access to the full API), and doesn't require timestamps. Whenever it sees a new host/plugin/type combination it adds it to a hash table and starts collecting the data over whatever time interval is specified in the config. When that time interval is up, pushes out one value (either the sum of the submitted values or their average, depending on what's needed).</div>
<div><br></div><div>Again due to my particular use case, it needed to be able to handle a large number of concurrent connections (50k potentially), so rather than spawning a thread per connection ala unixsock, I kick off one thread when the plugin init's and then run a libevent server inside that thread and do everything asynchronously.</div>
<div><br></div><div>So, then, two questions: First, did I just reinvent the wheel on this? Is there a plugin that already satisfies this sort of need? Second, if this is new, is anyone else interested in this sort of functionality? I'd be glad to hand off the code. The only tricky part is that it depends on libevent and, to a lesser extent, glib-2.0, which makes it a little less clean to build than the core functionality. </div>
<div><br></div><div>Thanks,</div><div>Shaun Lindsay</div>