[collectd] Strange behaviour of collectd with rrdcached

[Lindsay Holmwood]

I've seen this particular error when rrdcached's basedir hasn't been
correctly set with -b, and strict path checking has been enabled with -B.

If you strace -f the rrdcached process, you'll see a bunch of output that
looks like this:

[pid 27926] poll([{fd=6, events=POLLIN|POLLPRI}], 1, 500) = 1 ([{fd=6,
revents=POLLIN}])
[pid 27926] read(6, "update
/var/lib/collectd/rrd/hostname/swap/swap_io-in.rrd 1429757745:0\n", 8192) =
95
[pid 27926] write(6, "-1 Permission denied\n", 21) = 21

Doing an lsof on the rrdcached process shows fd6 is socket rrdcached and
collectd are talking to each other over:

rrdcached 27879 root    6u  unix 0xffff8800e80a1880      0t0 2563083
/var/run/rrdcached.sock

And if you look in the collectd logs, you'll see a bunch of messages that
look like this:

[2015-04-23 02:53:52] rrdcached plugin: rrdc_update
(/var/lib/collectd/rrd/host/swap/swap_io-in.rrd,
[1429757723:369514700800.000000], 1) failed with status -1.

To fix this, ensure that the -b argument to rrdcached matches the DataDir
setting on the rrdcached plugin.

It's very possible your actual problem is different to the one I outlined
above.

The good news is rrdcached is definitely reporting something back to
collectd, given you're seeing a failure returned in collectd's rrdcached
plugin.

My general process for debugging issues with collectd + rrdcached is:

 0. Crack out strace and see what's happening inside rrdcached. Use -s 256
so you can see the full string rrdcached is pushing back to collectd.
 1. Eliminate rrdcached, and replicate behaviour with the collectd + the
 vanilla rrdtool plugin .
 2. If you can't replicate, the problem is with rrdcached.
 3. Crack out strace and see what's happening inside rrdcached.
 4. Connect to the socket rrdcached is listening on and poke at it with
PENDING, FLUSH, STATS.

Hope that helps,
Lindsay

On Wed, Apr 22, 2015 at 9:15 PM Sergey <a_s_y at sama.ru> wrote:

> On Friday 12 December 2014, Marc wrote:
>
> > rrdcached plugin: rrdc_update
> (/var/lib/collectd/rrd/test_collectd/cpu-0/cpu-interrupt.rrd,
> [1418377171:3440], 1) failed with status -1.
> > Filter subsystem: Built-in target `write': Dispatching value to all
> write plugins failed with status -1.
>
> I have a same problem.
>
> Apr 22 13:05:41 collectd[14593]: rrdcached plugin: rrdc_update
> (/var/lib/collectd/rrd/hostname/apcups/temperature.rrd,
> [1429693527:32.800000], 1) failed with status -1.
>
> I used -j option of rrdcached and looked to journal:
>
> update /var/lib/collectd/rrd/hostname/apcups/temperature.rrd
> 1429693516:32.800000
> update /var/lib/collectd/rrd/hostname/apcups/temperature.rrd
> 1429693536:32.800000
> update /var/lib/collectd/rrd/hostname/apcups/temperature.rrd
> 1429693527:32.800000 <--
> update /var/lib/collectd/rrd/hostname/apcups/temperature.rrd
> 1429693546:32.800000
> update /var/lib/collectd/rrd/hostname/apcups/temperature.rrd
> 1429693556:32.800000
>
> Attempt to write incorrect entry was really.
> This happens quite often for different hosts:
>
> # grep "failed with status \-1" messages|wc -l
> 123578
>
> Could this be a problem of rrdcached ?
>
> --
> Regards, Sergey
>
> _______________________________________________
> collectd mailing list
> collectd at verplant.org
> http://mailman.verplant.org/listinfo/collectd
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20150423/704d5ea6/attachment.html>