[collectd] patch for email stats from postfix and amavisd-new, dns stats from powerdns

[Luke Heberling]

Alexander Wirt wrote:
> Luke Heberling schrieb am Sonntag, den 18. November 2007:
>> amavisd-new and postfix statistics are gathered by parsing the
>> log file (i.e. /var/log/mail.log). Unfortunately there does not
>> appear to be a better way of doing this.
>>     
> Please don't do this. At least not for amavisd-new. Use amavisd-nanny or the
> bdb stats databases. Alternativly you can also call the snmp agent by hand.
>
> But do _not_ scan logfiles. The format is adaptable and will offen be
> changed. and also scanning logs is pervert and something that should never be
> done. 
Presumably you mean that the *automated* scanning of logs is perverse. I 
would like to respectfully take issue with that. I may not have an 
entirely objective point of view due to a penchant for tools like 
logcheck and fail2ban, but I think I have some strong arguments.

Implicit in your first statement is that scanning the logs for postfix 
messages is acceptable, though probably far from ideal. Is this a 
correct interpretation? I've yet to see an alternative and programs like 
pflogsumm and mailgraph seem pretty popular.

Log formats are as you say, adaptable, but I would say that very few 
that I've run across change 'often'. Amavisd-new log format in 
particular has changed slightly but is quite structured, especially 
compared to postfix.

You also mention the bdb stats maintained by amavisd-new (and the snmp 
agent which uses the bdb). This is a valid point. This is a good way to 
get detailed statistics, particularly in the recent versions of 
amavisd-new as it has matured. Modifying amavis-agent into a perl plugin 
for collectd would probably be pretty simple. But it too has the same 
pitfalls as the logfile: the format changes incompatibly. in the 2.5.0 
release earlier this year such a change was made, and the amavis-agent 
program had to be changed. This would have broken any collectd plugin 
using the bdb in the same way as changing the log format would break my 
plugin.

I encourage you to take a look at the sheer number of log parsers that 
are linked directly from http://www.ijs.si/software/amavisd/#contrib. I 
count seven different log parsers that do various things. At least two 
of them are specifically for creating rrd files (though not nearly as 
efficient I daresay as collectd). I see no links to programs which use 
the bdb database.  The amavis-agent program is entirely within the 
distribution, so changes to the bdb format could be more easily 
justified as having less impact on third party programs.  If the snmp 
plugin for collectd were used to communicate with amavis-agent this 
would remove the
dependency on the bdb format but require the amavisd-new snmp agent to 
be running all the time.

Given that parsing the logs is clearly endorsed by the distribution and 
the sheer volume of programs that rely on the structured format, it 
seems that they would weigh heavily any further changes. Should they 
decide to go forward with such changes, this plugin would have to be 
updated. I think that if the community had a use for an updated version 
of the plugin, updating it would be a pretty trivial exercise. Plus I'd 
be perfectly willing to make such updates into the foreseeable future.

On the other hand, if you guys would like the powerdns plugin by itself 
I'd be happy to create that patch.  Maybe the amafix plugin could find 
its way into the contrib/ section or something.

Luke Heberling