[collectd] Collectd with rrdcached consumes all system memory

Erasmus Oblar eoblar at gmail.com
Wed Aug 14 22:22:49 CEST 2013

Hi.  I'm running into an odd problem when using collectd with the
rrdcached plugin.

Collectd 5.2.1, running on Debian 7, with rrdtool 1.4.7
Hardware: HP DL360 G7 w/48 GB memory and an intel 910 SSD that
holds ~400K rrd files.

Plugin config info:

<Plugin "rrdcached">
  DaemonAddress "unix:/var/run/rrdcached.sock"
  DataDir "/net/rrds2"
  CreateFiles true
  RRARows 12960
  RRATimespan 129600
  RRATimespan 777600
  RRATimespan 7776000
  RRATimespan 23328000

rrdcached options:
-b /net/rrds2 -s www-data -m 664 -l /var/run/rrdcached.sock -w 300 -t 2

When I run collectd with the rrdcached plugin, the collectd process
consumes all server memory and eventually gets whacked by the OOM killer.
rrdcached process never gets larger than a few hundred megabytes and when I
at the statistics the queue is almost always 0 in size (it does grow a bit
the write window, but goes back to zero quickly thereafter.)

I've also run rrdcached with and without a journal, and the journal files on
regular disk and on a ramdisk - it makes no real difference.  I've also
with more write threads (up to 10), but the differences are minimal, and 2
threads seem to perform the best.

If I dump the rrdcached plugin and use the rrdtool plugin - no problems.
collectd process will grow to a little over 1 GB in size and pretty much
that way.  This is a less than ideal solution, becuase there's no mechanism
flush pending data to the rrds for 'live' status.

Any ideas on how I could debug this further?  Reading around a bit, some
mentioned that collectd might not be pushing data to rrdcached fast enough,
is holding the data in memory.  That doesn't seem to be the case to me,
it looks like the rrd files are being kept current.  I'm at a bit of a loss
to how to proceed.  Thanks in advance for any ideas.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20130814/b6d68516/attachment.html>

More information about the collectd mailing list