[collectd] Simple Tail plugin regex -- count lines

[Bob.Boldin at cox.com]

Martin,

  The part I struggled with at first is the data is going to be converted into a rate - so the number represents Transactions per second.  I found on my use of tail I needed to increase the polling interval so that enough transactions were caught between polls to actually create meaningful data points.

In the global section I went to:
Interval 300

10,000 records in an hour would still only be 3 tps

In my tail match section using:
      DSType "CounterInc"
      Type "invocations"

Hope it helps

Bob

-----Original Message-----
From: collectd-bounces at verplant.org [mailto:collectd-bounces at verplant.org] On Behalf Of Martin B. Smith
Sent: Monday, March 14, 2011 11:54 AM
To: collectd at verplant.org
Subject: [collectd] Simple Tail plugin regex -- count lines

Hi all,

I've been struggling with the regex and/or dstype of the tail plugin. 
I'm basically wanting to count lines in an audit file that logs 
authentications. Any line should match. Here's what I've got:

LoadPlugin "tail"
<Plugin "tail">
   <File "/var/log/shibboleth-idp/idp-audit.log">
     Instance "idp_audit_log"
     <Match>
       Regex ".*"
       DSType "CounterInc"
       Type "derive"
       Instance "urn_ufl_edu"
     </Match>
   </File>
</Plugin>

I know that the log has more lines in it, since I can run this command 
and see the line count jump dramatically even in just a few seconds:

$ cat /var/log/shibboleth-idp/idp-audit.log  | wc -l

Unfortunately, when I dump the rrd, I get AVERAGE values like:

<!-- 2011-03-14 11:15:00 EDT / 1300115700 --> <row><v> 1.9156000000e+00 
</v></row>
<!-- 2011-03-14 11:20:00 EDT / 1300116000 --> <row><v> 1.9638666667e+00 
</v></row>
<!-- 2011-03-14 11:25:00 EDT / 1300116300 --> <row><v> 2.1036444444e+00 
</v></row>
<!-- 2011-03-14 11:30:00 EDT / 1300116600 --> <row><v> 1.9892000000e+00 
</v></row>
<!-- 2011-03-14 11:35:00 EDT / 1300116900 --> <row><v> 1.9534666667e+00 
</v></row>

I *know* I'm seeing more than 1-2 lines in a 5 minute interval. Is it 
possible the numbers are too large that I'm substracting, yielding an 
overflow? A single hour can sometimes have 10k lines in the log file.

Thanks in advance for your consideration!
-- 
Martin B. Smith
smithmb at ufl.edu - (352) 273-1374
CNS/Open Systems Group
University of Florida