[collectd] exec plugin stuck on mutex

[Ryan Tomayko]

On Tue, Mar 9, 2010 at 12:03 PM, Florian Forster <octo at verplant.org> wrote:
> Hi again,
>
> On Tue, Mar 09, 2010 at 11:46:20AM -0800, Ryan Tomayko wrote:
>> # lsof -p 25335
>> […]
>> collectd 25335 root    0r  FIFO    0,6         1285434092 pipe
>> collectd 25335 root    1w  FIFO    0,6         1285434093 pipe
>> collectd 25335 root    2w  FIFO    0,6         1285434094 pipe
>>
>> Not much useful in there that I can see.
>
> actually, this proves that this is in fact the forked child. Since the
> network socket is not open, it means that the filehandles (except the
> standard ones) have been closed and the pipes have been dup'ed to
> standard input, output and error.
>
> The next interesting bit would be the content of "/proc/$PID/status" for
> such a weird child. Maybe the signal mask or UID/GID can tell me which
> step would be the next and might be the failing one.

You got it:

# cat /proc/25335/status
Name:   collectd
State:  S (sleeping)
Tgid:   25335
Pid:    25335
PPid:   19346
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 64
Groups: 0
VmPeak:   227240 kB
VmSize:   227240 kB
VmLck:         0 kB
VmHWM:       796 kB
VmRSS:       796 kB
VmData:   188820 kB
VmStk:        84 kB
VmExe:       148 kB
VmLib:      2264 kB
VmPTE:       128 kB
Threads:        1
SigQ:   0/139264
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000
SigCgt: 0000000180014202
CapInh: 0000000000000000
CapPrm: ffffffffffffffff
CapEff: ffffffffffffffff
CapBnd: ffffffffffffffff
Cpus_allowed:   000000ff
Cpus_allowed_list:      0-7
Mems_allowed:   00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        1
nonvoluntary_ctxt_switches:     0

Here's the process hierarchy for that guy for context (25335 will
orphan and hang around when I restart with the init.d script):

# pstree -apu 19346

collectd,19346 -C /etc/collectd/collectd.conf -f
  ├─collectd,25335 -C /etc/collectd/collectd.conf -f
  ├─collectd,29459 -C /etc/collectd/collectd.conf -f
  ├─{collectd},19347
  ├─{collectd},19348
  ├─{collectd},19349
  ├─{collectd},19350
  ├─{collectd},19351
  ├─{collectd},25334
  └─{collectd},29458

>> I'll give that a shot. Thanks a ton for all your help here.
>
> You're welcome ;) I'm astonished again and again how hard forking a
> process "right" actually is ;)

Indeed. Definitely plenty of rope there. Still, I love it :)

Thanks,
Ryan