[collectd] Custom plugin for spam statistics

[Mirko Buffoni]

At 11:04 12/03/2009 +0100, Florian Forster wrote:
>Hi Mirko,
>
>On Thu, Mar 12, 2009 at 09:49:03AM +0100, Mirko Buffoni wrote:
> > I'd like to add a custom plugin to synthesyze my antispam statistics
> > into collectd.
> >
> > I'm using a custom made qmail-based solution, with a set of rules and
> > programs to intercept spam and viruses.
> > All notices are sent to qmail multilog, which automatically rotates
> > when log reaches a certain size.
> > The number of rotated logs vary from day to day.  At the end of the
> > day, logs are combined into a big log on another machine, and parsed
> > to gather statistics.
> > This stage could be enough to obtain all the information needed, but
> > I'm losing realtime.  I can live with that, but just out of curiosity,
> > I'd like to know if realtime can be obtained in some way.
> >
> > Do you think this is the only applicable solution, or is there any
> > smarter way to parse logs in realtime to updated collectd rrd files?
> > I thought of tail plugin as a possible way, but I'm not too confident
> > with that plugin.
>
>yes, the `tail' plugin would be the way to go here. Why do you feel
>uncomfortable with that plugin?

I feel unconfortable because:

- qmail log file have timestamp expressed in tai64 format.  Tail plugin should
   use a timestamp to put data into rrd database, but I think it cannot read
   tai64 dates (haven't digged source yet, so I may be wrong)
- how does tail plugin works?  does it rescan the entire log?  if not how
   does it know how much tail too keep from the log file?
- if rescan, shouldn't this overhead too much monitored machine?
- I want to build a cumulative graph (like dns queries) with different colors
   for each spam/virus/legit mail I get.  Can this be done with tail plugin?

Mirko