Hello,<div><br></div><div>I've checked the collectd source code from 'collectd-5.0' branch in git. </div><div>and made some patch to relieve the memory leak symptoms.</div><div>The solution was all about releasing the reference count of 'new reference' python object in python extension.</div>
<div>There is wonderful document about python extension. <a href="http://edcjones.tripod.com/refcount.html">http://edcjones.tripod.com/refcount.html</a></div><div><br></div><div>I've done several tests to see it resolves the leak. </div>
<div>following is the result and I guess it works.</div><div> </div><div># case when the collectd plugin keeps works nomaly without any error. </div><div> (memory chages after keep running collectd for 16 Hours)</div>
<div>case #1 (based on 'collectd-5.0' branch) : 8140 KB => 17596 KB</div><div>case #2 (based on 'collectd-5.0' branch with the patch) : 7672 KB => 7728 KB</div><div><br></div>
<div><br></div><div># case when the collectd plugin keeps fails in the business logic (see attachment) </div><div> (memory chages after keep running collectd for 16 Hours)</div>
<div>case #3 (based on 'collectd-5.0' branch) : 10728 KB => 68560 KB</div><div>case #4 (based on 'collectd-5.0' branch with the patch) : 8060 KB => 8184 KB </div>
<div><br></div><div>I would like to contribute my patch and experiences.</div><div>Please check my attachment. </div><div><br></div><div>Thank you</div><div> </div><div>HTH.</div><div>james.kim</div><div>NexR corp, The Best Big Data Solution provider.</div>
<div>Korea</div><div><br></div><div><br></div><div><br><div class="gmail_quote">2012³â 4¿ù 16ÀÏ ¿ÀÈÄ 12:23, James±è¹Î¼® <span dir="ltr"><<a href="mailto:james.kim@nexr.com">james.kim@nexr.com</a>></span>´ÔÀÇ ¸»:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><p>Hello,</p>
<p>I'm james.</p>
<p>Thank you for releasing the collectd.</p>
<p>I developed a python plugin for collected.</p>
<p>and experiencing memory consumption growth.</p>
<p>as time goes, memory size of collected process keeps growing.</p>
<p><br></p><p>I've searched similar cases with my case(<a href="http://www.mail-archive.com/collectd@verplant.org/msg01688.html" target="_blank"><span>http://www.mail-archive.com/collectd@verplant.org/msg01688.html</span></a>)</p>
<p>but not good for me with patched version.</p>
<p><br></p>
<p>this is my environment.</p>
<p>os: linux centos 5.5 Linux 2.6.18-128.1.6.el5xen x86_64 </p>
<p><span>collectd: collectd-5.0 branch. in git (<a href="https://github.com/octo/collectd/tree/collectd-5.0" target="_blank"><span>https://github.com/octo/collectd/tree/collectd-5.0</span></a>)</span></p>
<p>python: 2.6.2, 2.7.3</p>
<p><br></p>
<p>my plugin is simple, just creates a collectd packet and dispatch to collected server.(at the bottom of this mail)</p>
<p>and attached more complex plugin for zookeeper monitoring</p>
<p>my collectd collects every 5second.</p>
<p>and memory grows 4k~ 20kbytes every 5~ 60sec(random) in both case( simple and complex plugin.)</p>
<p><br></p>
<p><br></p>
<p>I've experimented several cases.</p>
<p>1. without python plugin, memory usage keeps steady ( 1.5M )</p>
<p>2. if python plugin is enabled, memory usage begins grow</p>
<p> with my plugin, 5064KB(initial)-> 5456(after 3hour)</p>
<p>3. strange thing is that just putting empty python configuration, </p>
<p> it still grows.</p>
<p><br></p>
<p><span style="background-color:rgb(204,204,204)"> LoadPlugin <span>python</span></span></p>
<p><span style="background-color:rgb(204,204,204)"> <Plugin <span>python</span>]></span></p>
<p><span style="background-color:rgb(204,204,204)"> ModulePath "/path/to/collectd_agent/share/<span>python</span>/"</span></p>
<p><span style="background-color:rgb(204,204,204)"> LogTraces true</span></p>
<p><span style="background-color:rgb(204,204,204)"> Interactive false</span></p>
<p><span style="background-color:rgb(204,204,204)"> </Plugin></span></p>
<p><br></p>
<p>4. it is not related to collected debug mode.(--enable-debug with recompile)</p>
<p> => no change. </p>
<p><br></p>
<p>5. the more python plugins are enabled, the more memory grows. </p>
<p> with 4 python plugins( with more complex plugins attached), it will grow 17Mbytes after 5hours. </p>
<p>6. if python plugin raises some exceptions, memory grows even faster.</p>
<p> with 4 python plugins( with more complex plugins), it will grow 72Mbytes after 5hours. </p>
<p><br></p>
<p>I've tried guppy(<a href="http://pypi.python.org/pypi/guppy/" target="_blank"><span>http://pypi.python.org/pypi/guppy/</span></a>) to track python memory leak.</p>
<p>but there is no symptom in python plugin itself.</p>
<p><br></p>
<p>Finally. I've used a valgrind(<a href="http://www.cprogramming.com/debugging/valgrind.html" target="_blank"><span>http://www.cprogramming.com/debugging/valgrind.html</span></a>) to spot problems using following command.</p>
<p>valgrind --tool=memcheck --suppressions=/python/path/valgrind-python.supp --leak-check=full <span>--error-limit=no </span><span>--log-file=valog ./sbin/collectd </span></p>
<p>if python plugin is disabled in configuration, there is no lost memory in report.</p>
<p>but if enabled python plugin ,it reports some following reports.</p>
<p><span> ( </span><span>to suppress warning, I've </span><span>recompiled python ( </span>--without-<span>pymalloc) and reinstalled collected )</span></p>
<p>I think that valgrind only reports about initial step of collected and unable to track the multi thread workers.</p>
<p> </p>
<p><br></p>
<p>Do you have any idea, what cause this problem?</p><p> thanks you for reading.</p>
<p><br></p>
<p><br></p>
<p>------------valgrind reports -----------------------------------------------------------------</p>
<p> 370 bytes in 1 blocks are possibly lost in loss record 2,847 of 3,541</p>
<p>==30213== at 0x4C2210C: malloc (vg_replace_malloc.c:195)</p>
<p>==30213== by 0x7CED83C: PyString_FromString (stringobject.c:143)</p>
<p>==30213== by 0x7D00A8F: PyType_Ready (typeobject.c:4059)</p>
<p>==30213== by 0x7CE3277: _Py_ReadyTypes (object.c:2132)</p>
<p>==30213== by 0x7D673C0: Py_InitializeEx (pythonrun.c:179)</p>
<p>==30213== by 0x7A4CE70: cpy_config (python.c:989)</p>
<p>==30213== by 0x4099C4: cf_read (configfile.c:361)</p>
<p>==30213== by 0x4062C8: main (collectd.c:477)</p>
<p><br></p>
<p><br></p>
<p>==7443== 44 bytes in 1 blocks are possibly lost in loss record 833 of 3,469</p>
<p>==7443== at 0x4C2276F: malloc (vg_replace_malloc.c:263)</p>
<p>==7443== by 0x7CEEB3E: PyString_FromStringAndSize (stringobject.c:88)</p>
<p>==7443== by 0x7D630CA: do_mkvalue (modsupport.c:427)</p>
<p>==7443== by 0x7D6382C: do_mktuple (modsupport.c:268)</p>
<p>==7443== by 0x7D6321B: do_mkvalue (modsupport.c:298)</p>
<p>==7443== by 0x7D639EB: va_build_value (modsupport.c:537)</p>
<p>==7443== by 0x7D63D6E: Py_BuildValue (modsupport.c:485)</p>
<p>==7443== by 0x7D6F0B2: _PySys_Init (sysmodule.c:1405)</p>
<p>==7443== by 0x7D6845F: Py_InitializeEx (pythonrun.c:215)</p>
<p>==7443== by 0x7A4DEC0: cpy_config (python.c:996)</p>
<p>==7443== by 0x4099C4: cf_read (configfile.c:361)</p>
<p>==7443== by 0x4062C8: main (collectd.c:477)<span> </span></p>
<p><br></p>
<p>==22090== 44 bytes in 1 blocks are possibly lost in loss record 843 of 3,581</p>
<p>==22090== at 0x4C2276F: malloc (vg_replace_malloc.c:263)</p>
<p>==22090== by 0x7CE57C8: PyObject_Malloc (obmalloc.c:943)</p>
<p>==22090== by 0x7CED87E: PyString_FromStringAndSize (stringobject.c:88)</p>
<p>==22090== by 0x7D620FA: do_mkvalue (modsupport.c:427)</p>
<p>==22090== by 0x7D6285C: do_mktuple (modsupport.c:268)</p>
<p>==22090== by 0x7D6224B: do_mkvalue (modsupport.c:298)</p>
<p>==22090== by 0x7D62A1B: va_build_value (modsupport.c:537)</p>
<p>==22090== by 0x7D62D9E: Py_BuildValue (modsupport.c:485)</p>
<p>==22090== by 0x7D6E4D2: _PySys_Init (sysmodule.c:1408)</p>
<p>==22090== by 0x7D6742F: Py_InitializeEx (pythonrun.c:222)</p>
<p>==22090== by 0x7A4BEA0: cpy_config (python.c:999)</p>
<p>==22090== by 0x409924: cf_read (configfile.c:361)</p>
<p><br></p>
<p><br></p>
<p>---------------- following is a simple python plugin ----------------------</p>
<p><br></p>
<p># Simple <span>Python</span> module </p>
<p>"""</p>
<p> Import python_plugin_test</p>
<p> <Module python_plugin_test></p>
<p> </p>
<p> </Module></p>
<p><br></p>
<p>"""</p>
<p><span>import</span> socket</p>
<p><span>import</span> collectd</p>
<p>import<span> gc</span></p>
<p><br></p>
<p><br></p>
<p><span>def</span> configer(ObjConfiguration):</p>
<p> collectd.debug(<span>'Configuring Stuff'</span>) </p>
<p> </p>
<p><br></p>
<p><span>def</span> initiator():</p>
<p> collectd.debug(<span>'</span><span>initing</span><span> stuff'</span>)</p>
<p> gc.set_debug(gc.DEBUG_LEAK)</p>
<p> </p>
<p><br></p>
<p><span>def</span> dispatcher(data=<span>None</span>):</p>
<p> metric = collectd.Values();</p>
<p><span> metric.plugin = </span>'python_plugin_test_counter'</p>
<p> metric.type = <span>'mysql_threads'</span></p>
<p> metric.values = [<span>5</span>,<span>10</span>,<span>15</span>,<span>10</span>]</p>
<p> metric.dispatch()</p>
<p><span> </span>#<span>gcresult</span>=gc.collect()</p>
<p><span> </span>#<a href="http://collectd.info" target="_blank">collectd.info</a>("python_plugin_test:gc:%s"%<span>gcresult</span>) </p>
<p> </p>
<p> </p>
<p> </p>
<p>#== Hook <span>Callbacks</span>, Order is important! ==#</p>
<p>collectd.register_config(configer)</p>
<p>collectd.register_init(initiator)</p>
<p>collectd.register_read(dispatcher)</p>
<p><br></p>
<p>---------------------------</p></div><span class="HOEnZb"><font color="#888888">-- <br><div>James.kim(±è¹Î¼®)</div><div>010 3266 8040</div><div><br></div><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>James.kim(±è¹Î¼®)</div><div>010 3266 8040</div><div><br></div><br>
</div>