[collectd] collectd) memory usage growing in python plugin.

Mon Apr 16 05:23:05 CEST 2012

Hello,

I'm james.

Thank you for releasing the collectd.

I developed a python plugin for collected.

and experiencing memory consumption growth.

as time goes, memory size of collected process keeps growing.

I've searched similar cases with my case(
http://www.mail-archive.com/collectd@verplant.org/msg01688.html)

but not good for me with patched version.

this is my environment.

os: linux centos 5.5 Linux  2.6.18-128.1.6.el5xen x86_64

collectd: collectd-5.0 branch. in git (
https://github.com/octo/collectd/tree/collectd-5.0)

python: 2.6.2, 2.7.3

my plugin is simple, just creates a collectd packet and dispatch to
collected server.(at the bottom of this mail)

and attached more complex plugin for zookeeper monitoring

my collectd collects every 5second.

and memory grows 4k~ 20kbytes every  5~ 60sec(random)  in both case( simple
and complex plugin.)

I've experimented several cases.

1. without python plugin, memory usage keeps steady ( 1.5M )

2. if python plugin is enabled, memory usage begins grow

    with my plugin, 5064KB(initial)-> 5456(after 3hour)

3. strange thing is that just putting empty python configuration,

   it still grows.

 LoadPlugin python

 <Plugin python]>

  ModulePath "/path/to/collectd_agent/share/python/"

  LogTraces true

  Interactive false

 </Plugin>

4.  it is not related to collected debug mode.(--enable-debug with
recompile)

     => no change.

5. the more python plugins are enabled, the more memory grows.

     with 4 python plugins( with more complex plugins attached), it will
grow 17Mbytes after 5hours.

6. if python plugin raises some exceptions, memory grows even faster.

     with 4 python plugins( with more complex plugins), it will grow
72Mbytes after 5hours.

I've tried guppy(http://pypi.python.org/pypi/guppy/) to track python memory
leak.

but there is no symptom in python plugin itself.

Finally. I've used a valgrind(
http://www.cprogramming.com/debugging/valgrind.html) to spot problems using
following command.

valgrind --tool=memcheck --suppressions=/python/path/valgrind-python.supp
--leak-check=full --error-limit=no  --log-file=valog ./sbin/collectd

if  python plugin is disabled in configuration, there is no lost memory in
report.

but if enabled python plugin ,it reports some following reports.

   ( to suppress warning, I've recompiled python ( --without-pymalloc) and
reinstalled collected )

I think that valgrind only reports  about initial step of collected and
unable to track the multi thread workers.

Do you have any idea, what cause this problem?

 thanks you for reading.

------------valgrind reports
-----------------------------------------------------------------

 370 bytes in 1 blocks are possibly lost in loss record 2,847 of 3,541

==30213==    at 0x4C2210C: malloc (vg_replace_malloc.c:195)

==30213==    by 0x7CED83C: PyString_FromString (stringobject.c:143)

==30213==    by 0x7D00A8F: PyType_Ready (typeobject.c:4059)

==30213==    by 0x7CE3277: _Py_ReadyTypes (object.c:2132)

==30213==    by 0x7D673C0: Py_InitializeEx (pythonrun.c:179)

==30213==    by 0x7A4CE70: cpy_config (python.c:989)

==30213==    by 0x4099C4: cf_read (configfile.c:361)

==30213==    by 0x4062C8: main (collectd.c:477)

==7443== 44 bytes in 1 blocks are possibly lost in loss record 833 of 3,469

==7443==    at 0x4C2276F: malloc (vg_replace_malloc.c:263)

==7443==    by 0x7CEEB3E: PyString_FromStringAndSize (stringobject.c:88)

==7443==    by 0x7D630CA: do_mkvalue (modsupport.c:427)

==7443==    by 0x7D6382C: do_mktuple (modsupport.c:268)

==7443==    by 0x7D6321B: do_mkvalue (modsupport.c:298)

==7443==    by 0x7D639EB: va_build_value (modsupport.c:537)

==7443==    by 0x7D63D6E: Py_BuildValue (modsupport.c:485)

==7443==    by 0x7D6F0B2: _PySys_Init (sysmodule.c:1405)

==7443==    by 0x7D6845F: Py_InitializeEx (pythonrun.c:215)

==7443==    by 0x7A4DEC0: cpy_config (python.c:996)

==7443==    by 0x4099C4: cf_read (configfile.c:361)

==7443==    by 0x4062C8: main (collectd.c:477)

==22090== 44 bytes in 1 blocks are possibly lost in loss record 843 of 3,581

==22090==    at 0x4C2276F: malloc (vg_replace_malloc.c:263)

==22090==    by 0x7CE57C8: PyObject_Malloc (obmalloc.c:943)

==22090==    by 0x7CED87E: PyString_FromStringAndSize (stringobject.c:88)

==22090==    by 0x7D620FA: do_mkvalue (modsupport.c:427)

==22090==    by 0x7D6285C: do_mktuple (modsupport.c:268)

==22090==    by 0x7D6224B: do_mkvalue (modsupport.c:298)

==22090==    by 0x7D62A1B: va_build_value (modsupport.c:537)

==22090==    by 0x7D62D9E: Py_BuildValue (modsupport.c:485)

==22090==    by 0x7D6E4D2: _PySys_Init (sysmodule.c:1408)

==22090==    by 0x7D6742F: Py_InitializeEx (pythonrun.c:222)

==22090==    by 0x7A4BEA0: cpy_config (python.c:999)

==22090==    by 0x409924: cf_read (configfile.c:361)

---------------- following is a simple python plugin ----------------------

# Simple Python module

"""

  Import python_plugin_test

  <Module python_plugin_test>

  </Module>

"""

import socket

import collectd

import gc

def configer(ObjConfiguration):

   collectd.debug('Configuring Stuff')

def initiator():

    collectd.debug('initing stuff')

    gc.set_debug(gc.DEBUG_LEAK)

def dispatcher(data=None):

    metric = collectd.Values();

    metric.plugin = 'python_plugin_test_counter'

    metric.type = 'mysql_threads'

    metric.values = [5,10,15,10]

    metric.dispatch()

    #gcresult=gc.collect()

    #collectd.info("python_plugin_test:gc:%s"%gcresult)

#== Hook Callbacks, Order is important! ==#

collectd.register_config(configer)

collectd.register_init(initiator)

collectd.register_read(dispatcher)

---------------------------
-- 
James.kim(김민석)
010 3266 8040
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20120416/59fd8a45/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zookeeper_stat_plugin.py
Type: application/octet-stream
Size: 7064 bytes
Desc: not available
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20120416/59fd8a45/attachment.obj>