[collectd] Collectd write_graphite plugin send NaN values every 20 seconds

[Giovanni Torres]

On Tue, Jan 26, 2016 at 10:57 PM Joel Serrano <joel at gogii.net> wrote:

> Hi,
>
> I'm using collectd's tail plugin to read from a logfile and send metrics
> to influxdb via write_graphite plugin.
>
> The strange thing is that when there are no new events in the logfile,
> collectd still send metrics with "nan" value, every 20 seconds.
>
> This makes influxdb complain:
>
> [graphite] 2016/01/26 19:43:47 unable to parse line:
> stats.server.13cn37.tail.fs.gauge.metric1 nan 1453866221: field
> "stats.server.13cn37.tail.fs.gauge.metric1" value: 'NaN" is unsupported
>
>
> My settings are very very simple:
>
> 1) collectd.conf: Load tail and write_graphite plugins.
>
>
> 2) The config for the write_graphite plugin:
>
> <Plugin "write_graphite">
>  <Node "cpd1">
>    Host "graphite.example.com
>    Port "2003"
>    Prefix "stats.server."
>    #Postfix ""
>    Protocol "tcp"
>    LogSendErrors true
>    EscapeCharacter "."
>    SeparateInstances true
>    StoreRates false
>    AlwaysAppendDS false
>  </Node>
> </Plugin>
>
>
> 3) The config for the tail plugin:
>
> <Plugin "tail">
>   <File "/usr/local/app/log/application.log">
>     Instance "fs"
>     <Match>
>       Regex "^.*metric_name: metric1, metric_value: (.*)$"
>       DSType "GaugeLast"
>       Type "gauge"
>       Instance "metric1"
>     </Match>
>
>     <Match>
>       Regex "^.*metric_name: metric2, metric_value: (.*)$"
>       DSType "GaugeLast"
>       Type "gauge"
>       Instance "metric2"
>     </Match>
>
>     <Match>
>       Regex "^.*metric_name: metric3, metric_value: (.*)$"
>       DSType "GaugeLast"
>       Type "gauge"
>       Instance "metric3"
>     </Match>
>
>     <Match>
>       Regex "^.*metric_name: metric4, metric_value: (.*)$"
>       DSType "GaugeLast"
>       Type "gauge"
>       Instance "metric4"
>     </Match>
>
>   </File>
> </Plugin>
>
>
>
> If there are new events in the logfile and the regex match, the values are
> sent correctly, if the values don't match or there are no new events, then
> they are all sent with "nan".
>
> Any idea what can be happening?
>
> InfluxDB: v0.9.5
> collectd: v5.4.1
>

I don't get this error. Which log file are you seeing this error?  I am
using collectd 5.5.0 and influxdb 0.9.6.1.  Perhaps you need an upgrade to
one or both.  You may want to peruse the changelogs to see this was a known
problem that has since been fixed.

I have a minimal config in collectd to match yours:

LoadPlugin tail
<Plugin tail>
 <File "/var/log/secure">
   Instance "secure"
   <Match>
      Regex "\\<sshd[^:]*: Invalid user [^ ]+ from\\>"
      DSType "CounterInc"
      Type "counter"
      Instance "sshd-invalid_user"
    </Match>
  </File>
</Plugin>

This was an example from the wiki:
https://collectd.org/wiki/index.php/Plugin:Tail#Invalid_SSH_login_attempts

This is a log file on a private VM that is not getting any active or failed
sshd attempts, so each tail interval should return zero.  I don't see any
errors in collectd.log or influxd.log.

Here is the output I get from influxdb:
> select * from
"stats.server.localhost.tail.secure.counter.sshd-invalid_user"
name: stats.server.localhost.tail.secure.counter.sshd-invalid_user
------------------------------------------------------------------
time value
1454180922000000000 0
1454180932000000000 0
1454180942000000000 0
1454180952000000000 0
1454180962000000000 0
...

No NaN's, just zeroes.

Hope that helps.

Giovanni
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20160130/c9428468/attachment.html>