[collectd] Collectd with rrdcached consumes all system memory

[Erasmus Oblar]

Hi.  I'm running into an odd problem when using collectd with the
rrdcached plugin.

Setup:
Collectd 5.2.1, running on Debian 7, with rrdtool 1.4.7
Hardware: HP DL360 G7 w/48 GB memory and an intel 910 SSD that
holds ~400K rrd files.

Plugin config info:

<Plugin "rrdcached">
  DaemonAddress "unix:/var/run/rrdcached.sock"
  DataDir "/net/rrds2"
  CreateFiles true
  RRARows 12960
  RRATimespan 129600
  RRATimespan 777600
  RRATimespan 7776000
  RRATimespan 23328000
</Plugin>

rrdcached options:
-b /net/rrds2 -s www-data -m 664 -l /var/run/rrdcached.sock -w 300 -t 2

When I run collectd with the rrdcached plugin, the collectd process
eventually
consumes all server memory and eventually gets whacked by the OOM killer.
 The
rrdcached process never gets larger than a few hundred megabytes and when I
look
at the statistics the queue is almost always 0 in size (it does grow a bit
during
the write window, but goes back to zero quickly thereafter.)

I've also run rrdcached with and without a journal, and the journal files on
regular disk and on a ramdisk - it makes no real difference.  I've also
tested
with more write threads (up to 10), but the differences are minimal, and 2
write
threads seem to perform the best.

If I dump the rrdcached plugin and use the rrdtool plugin - no problems.
 The
collectd process will grow to a little over 1 GB in size and pretty much
stay
that way.  This is a less than ideal solution, becuase there's no mechanism
to
flush pending data to the rrds for 'live' status.

Any ideas on how I could debug this further?  Reading around a bit, some
people
mentioned that collectd might not be pushing data to rrdcached fast enough,
and
is holding the data in memory.  That doesn't seem to be the case to me,
because
it looks like the rrd files are being kept current.  I'm at a bit of a loss
as
to how to proceed.  Thanks in advance for any ideas.

EO
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20130814/b6d68516/attachment.html>