[collectd] output plugin for Splunk

[Florian Forster]

Hi Clay,

On Tue, Aug 19, 2008 at 04:31:31PM -0700, Clay Loveless wrote:
> I'm not expecting to find Splunk experts on this list [...]

I'm afraid this is the first time I've heard about it. I love the name
and the company's motto (``Take the sh out of IT''), though :)

> I'm interested in a collectd output plugin to pipe collected
> information directly into Splunk.

Hm, the documentation at [0] and [1] is very vague about how UDP inputs
are handled. They write something about acting as a syslog daemon, but
that's about it..

> - created a Splunk UDP data input on port 3012, sourcetype "misc_text"

If I understand [2] correctly, Splunk will try to automatically
determine the ``source type'' of those values and create a new one if
none of the existing types fit.

> So, it appears that the message is making it to my Splunk box, but
> because the basic UDP listener on that side does not have the
> parse_packet() fu that network.c has, it just barfs and tosses the  
> packet.

Well, it all looks like it's expecting some text based format. The
network protocol used by collectd is a binary protocol, though. Also,
the information is slightly ``compressed'', requiring a ``state'' when
parsing the package. I doubt that any software can automagically handle
that.

> Is there anything I could/should be doing differently here?

From what I've seen I'd say there are two possibilities:
1) Since on [2] they state
   ``If Splunk fails to recognize a common format, or applies an
     incorrect source type value, you should report the problem to
     Splunk support and send us a sample file.''
   you might just open a bug report with Splunk. After all, they claim
   on [3] that
   ``If a machine can generate it - Splunk can eat it.''
2) Write a new output plugin for collectd which sends data to Splunk in
   a format it understands or is able to learn. A line based ASCII
   protocol would have a good chance from what I gather.

Other that that I'm afraid I'm totally clueless.. Hope this helps,
though..

Regards,
-octo

[0] <http://www.splunk.com/doc/3.3.1/admin/InputConfig>
[1] <http://www.splunk.com/doc/3.3.1/admin/inputsconfspec>
[2] <http://www.splunk.com/doc/3.3.1/admin/SourceTypes>
[3] <http://www.splunk.com/company>
-- 
Florian octo Forster
Hacker in training
GnuPG: 0x91523C3D
http://verplant.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mailman.verplant.org/pipermail/collectd/attachments/20080820/01a6b1ec/attachment.pgp