[collectd] Python plugin threading behavior question

[Jevgenij Tsoi]

Hello,

First of all, thank you for a great metrics collection framework!

I am using collectd 4.10.1 on Ubuntu 12.04

I hope this is the right forum to ask a question about the python plugin.
We are writing a custom plugin that will read some internal stats from
within several web-workers running on the local machine.

Looking at documentation, there is an explicit warning that the python
plugins must be thread safe. Also,
several plugins written in python (one being collectd-carbon) explicitly
use a threading.Lock() to do some operations.

Looking at logs, for read-only plugins, it seems that only one thread is
calling the read callback at a time. (Even with multiple read-threads and
explicit sleep in read callback longer than the interval)

So my 2 questions are:

In practice, only write plugins need proper locking and I can still count
on the GIL in all cases?
(Meaning, only complex non atomic operations need to be synchronized)

And finally, the python module seems to be imported twice?
(At first I thought it once per read-thread, because i had 2, but it is
regardless of threads)
The module is loaded twice, the configure callback is called twice, BUT the
initialize-callback is called only once.  Why is it behaving this way?

Configuration:

> LoadPlugin python
> <Plugin python>
>   ModulePath "/home/coco/tmp"
>   LogTraces true
>   Interactive false
>   Import cherrypy_plugin
>   <Module cherrypy_plugin>
>     Hello "These" "are" "the" "inputs"
>   </Module>
> </Plugin>


Log output:

> [2012-12-20 11:49:34] [info] Loading cherrypy plugin
> [2012-12-20 11:49:34] [info] Loaded cherrypy plugin
> [2012-12-20 11:49:34] [info] Instance of cherrypy plugin created
> <cherrypy_plugin.CherryPyCollector object at 0xb607b22c>
> [2012-12-20 11:49:34] [info] Configuring cherrypy plugin
> [2012-12-20 11:49:34] [info] Configuration Hello: ('These', 'are', 'the',
> 'inputs')
> [2012-12-20 11:49:34] [info] Loading cherrypy plugin
> [2012-12-20 11:49:34] [info] Loaded cherrypy plugin
> [2012-12-20 11:49:34] [info] Instance of cherrypy plugin created
> <cherrypy_plugin.CherryPyCollector object at 0xb61d71cc>
> [2012-12-20 11:49:34] [info] Configuring cherrypy plugin
> [2012-12-20 11:49:34] [info] Configuration Hello: ('These', 'are', 'the',
> 'inputs')
> [2012-12-20 11:49:34] [info] Initializing cherrypy plugin
> [2012-12-20 11:49:34] [info] Threading module: 3055395796
> [2012-12-20 11:49:34] [info] Instance of cherrypy plugin initialized
> <cherrypy_plugin.CherryPyCollector object at 0xb61d71cc>
> [2012-12-20 11:49:34] [info] Initialization complete, entering read-loop.
> [2012-12-20 11:49:34] [info] Read invoked on cherrypy plugin
> <cherrypy_plugin.CherryPyCollector object at 0xb61d71cc> on thread Dummy-1
> [2012-12-20 11:49:36] [info] Read invoked on cherrypy plugin
> <cherrypy_plugin.CherryPyCollector object at 0xb61d71cc> on thread Dummy-1
> [2012-12-20 11:49:37] [info] Read invoked on cherrypy plugin
> <cherrypy_plugin.CherryPyCollector object at 0xb61d71cc> on thread Dummy-1




Thanks in advance,

/JT at Virtusize
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20121220/91feaf49/attachment-0001.html>