[collectd] Fwd: client is never sending data to server

Lonni J Friedman netllama at gmail.com
Tue Jul 31 04:00:37 CEST 2018


On Mon, Jul 30, 2018 at 1:24 PM Ricardo J. Barberis
<ricardo at palmtx.com.ar> wrote:
>
> Hello!
>
> El Domingo 29/07/2018 a las 17:48, Lonni J Friedman escribió:
> > Hello,
> > I'm in the process of setting up collectd (5.8.0).  I'm using influxdb
> > and grafana to round out the metrics pipeline.  I have everything
> > working ok on the 'server'.  However, attempting to add a client
> > (which sends its metrics to the collectd server) is not working, and
> > I'm confused why its failing to work.  When I say its not working, I
> > mean that collectd on the client doesn't appear to be sending anything
> > to the collectd on the server.
> >
> > On the server (10.0.0.1), the network plugin section looks like this:
> >
> > <Plugin network>
> >     Server "127.0.0.1" "8096"
> >     <Listen "0.0.0.0" "25826">
> >         SecurityLevel Sign
> >         AuthFile "/etc/collectd.d/passwd"
> >         Interface "enp5s0"
> >     </Listen>
> >     MaxPacketSize 1452
> > </Plugin>
> >
> >
> > On the client,(10.0.0.9) the network plugin section looks like this:
> >
> > <Plugin network>
> >         <Server "10.0.0.1" "25826">
> >                 SecurityLevel Encrypt
> >                 Username "netllama"
> >                 Password "nopenopenope"
> >         </Server>
> >         MaxPacketSize 1452
> > </Plugin>
>
> IUUC, you need to use Sign or Encrypt on both, client and server.

Ahh, good catch.  I made the change, and restarted client & server,
but didn't notice any difference.

>
> Another thing to try is not to set MaxPacketSize, it never worked for me but
> it's been a while since I tested it.

Tried that, but no difference.  I do periodically get some warnings
when its not set:
network plugin: parse_packet: Received truncated packet, try
increasing `MaxPacketSize'

>
>
> > On the client, I see the following output when collectd is started:
> >
> > [2018-07-29 13:43:54] [info] plugin_load: plugin "cpu" successfully loaded.
> > [2018-07-29 13:43:54] [info] plugin_load: plugin "disk" successfully
> > loaded. [2018-07-29 13:43:54] [info] plugin_load: plugin "interface"
> > successfully loaded.
> > [2018-07-29 13:43:54] [info] plugin_load: plugin "load" successfully
> > loaded. [2018-07-29 13:43:54] [info] plugin_load: plugin "memory"
> > successfully loaded. [2018-07-29 13:43:54] [info] plugin_load: plugin
> > "network" successfully loaded. [2018-07-29 13:43:54] [info] plugin_load:
> > plugin "uptime" successfully loaded. [2018-07-29 13:43:54] [notice] Systemd
> > detected, trying to signal readyness. [2018-07-29 13:43:54] [info]
> > Initialization complete, entering read-loop.
> >
> > and then there's no more output (errors, etc) after many minutes of
> > waiting.
> >
> > I tried running strace (everything is Linux) on the client's collectd
> > process, and the only output are a bunch of nanosleep (this is after
> > waiting more than 10 seconds):
> >
> > nanosleep({tv_sec=9, tv_nsec=999681119}, 0x7ec885e8) = 0
> > nanosleep({tv_sec=9, tv_nsec=999506135}, 0x7ec885e8) = 0
> > nanosleep({tv_sec=9, tv_nsec=999502288}, 0x7ec885e8) = 0
> > nanosleep({tv_sec=9, tv_nsec=999463651}, 0x7ec885e8) = 0
> > nanosleep({tv_sec=9, tv_nsec=999401109}, 0x7ec885e8) = 0
> > nanosleep({tv_sec=9, tv_nsec=999439759}, 0x7ec885e8) = 0
> > nanosleep({tv_sec=9, tv_nsec=999443827}, 0x7ec885e8) = 0
> > nanosleep({tv_sec=9, tv_nsec=999448938}, 0x7ec885e8) = 0
> > nanosleep({tv_sec=9, tv_nsec=999431915}, 0x7ec885e8) = 0
> >
> >
> > Shouldn't there be more than just nanosleep if the client was actually
> > doing something?  Am I missing some obvious solution?
> >
> > thanks!
>
> About strace, make sure to use 'strace -f' (follow children) in case the
> collecting an sending don't happen on the parent process.

Good point.  I do see more output once I follow the children, but nothing good:

[pid 12483] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 12483] futex(0x48416c, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 12483] open("/proc/stat", O_RDONLY|O_LARGEFILE) = 5
[pid 12483] fstat64(5, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid 12483] read(5, "cpu  721654 378461 1463321 54888"..., 1024) = 935
[pid 12483] read(5, "", 1024)           = 0
[pid 12483] close(5)                    = 0
[pid 12483] futex(0x484248, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 12480] <... futex resumed> )       = 0
[pid 12483] futex(0x484248, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12480] futex(0x4841fc, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12483] <... futex resumed> )       = 1
[pid 12479] <... futex resumed> )       = 0
[pid 12483] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12479] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12483] <... futex resumed> )       = 1
[pid 12480] <... futex resumed> )       = 0
[pid 12483] futex(0x484248, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12480] futex(0x4841fc, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12483] <... futex resumed> )       = 1
[pid 12479] <... futex resumed> )       = 0
[pid 12483] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12478] <... futex resumed> )       = 0
[pid 12479] futex(0x48424c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 12483] <... futex resumed> )       = 1
[pid 12480] <... futex resumed> )       = 0
[pid 12483] futex(0x484248, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12480] futex(0x4841fc, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12483] <... futex resumed> )       = 1
[pid 12476] <... futex resumed> )       = 0
[pid 12483] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12478] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12483] <... futex resumed> )       = 1
[pid 12480] <... futex resumed> )       = 0
[pid 12478] <... futex resumed> )       = 0
[pid 12483] futex(0x48424c, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12480] futex(0x4841fc, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12483] <... futex resumed> )       = 1
[pid 12478] futex(0x4841fc, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12483] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12478] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12483] <... futex resumed> )       = 1
[pid 12480] <... futex resumed> )       = 0
[pid 12477] <... futex resumed> )       = 0
[pid 12483] futex(0x48424c, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12480] futex(0x4841fc, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12483] <... futex resumed> )       = 1
[pid 12479] <... futex resumed> )       = 0
[pid 12483] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12484] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 12483] <... futex resumed> )       = 1
[pid 12484] futex(0x48416c, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12480] <... futex resumed> )       = 0
[pid 12484] <... futex resumed> )       = 0
[pid 12483] futex(0x48424c, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12484] open("/proc/diskstats", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid 12483] <... futex resumed> )       = 0
[pid 12480] futex(0x4841fc, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12479] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12484] <... open resumed> )        = 5
[pid 12483] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12484] fstat64(5,  <unfinished ...>
[pid 12483] <... futex resumed> )       = 0
[pid 12484] <... fstat64 resumed> {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid 12480] <... futex resumed> )       = 0
[pid 12479] <... futex resumed> )       = 1
[pid 12478] futex(0x4841fc, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12477] futex(0x4841fc, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12476] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12484] read(5,  <unfinished ...>
[pid 12482] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 12478] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12479] sendto(3,
"\0\0\0\rllamapi2\0\0\10\0\f\26\327\357\333\245\266Tx\0\t\0\f\0\0\0"...,
1334, 0, {sa_family=AF_INET, sin_port=htons(25826),
sin_addr=inet_addr("10.0.0.1")}, 16 <unfinished ...>
[pid 12478] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12477] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12476] <... futex resumed> )       = 0
[pid 12484] <... read resumed> "   1       0 ram0 0 0 0 0 0 0 0 "...,
1024) = 1024
[pid 12482] futex(0x48416c, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12480] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12478] <... futex resumed> )       = 0
[pid 12477] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12479] <... sendto resumed> )      = 1334
[pid 12476] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12485] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 12478] futex(0x76ad83c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12477] <... futex resumed> )       = 0
[pid 12485] futex(0x48416c, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 12485] open("/proc/loadavg", O_RDONLY <unfinished ...>
[pid 12484] read(5,  <unfinished ...>
[pid 12483] futex(0x4841b8,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1533001592,
tv_nsec=601069523}, 0xffffffff <unfinished ...>
[pid 12485] <... open resumed> )        = 6
[pid 12484] <... read resumed> "6426 240462 15519440 583375720 0"...,
1024) = 278
[pid 12485] read(6,  <unfinished ...>
[pid 12482] <... futex resumed> )       = 0
[pid 12485] <... read resumed> "0.05 0.05 0.00 4/138 12507\n", 64) = 27
[pid 12481] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 12485] close(6 <unfinished ...>
[pid 12481] futex(0x48416c, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12484] read(5,  <unfinished ...>
[pid 12482] open("/proc/net/dev", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid 12485] <... close resumed> )       = 0
[pid 12484] <... read resumed> "", 1024) = 0
[pid 12481] <... futex resumed> )       = 0
[pid 12480] <... futex resumed> )       = 0
[pid 12482] <... open resumed> )        = 6
[pid 12484] close(5 <unfinished ...>
[pid 12479] futex(0x76ad83c0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12485] futex(0x4841b8,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1533001602,
tv_nsec=587387929}, 0xffffffff <unfinished ...>
[pid 12481] open("/proc/meminfo", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid 12477] futex(0x76ad83c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12476] <... futex resumed> )       = 0
[pid 12481] <... open resumed> )        = 7
[pid 12477] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12482] fstat64(6,  <unfinished ...>
[pid 12484] <... close resumed> )       = 0
[pid 12482] <... fstat64 resumed> {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid 12484] futex(0x4841b8,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1533001602,
tv_nsec=593172013}, 0xffffffff <unfinished ...>
[pid 12482] read(6,  <unfinished ...>
[pid 12481] fstat64(7,  <unfinished ...>
[pid 12480] futex(0x76ad83c0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12481] <... fstat64 resumed> {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid 12482] <... read resumed> "Inter-|   Receive               "...,
1024) = 583
[pid 12481] read(7,  <unfinished ...>
[pid 12480] <... futex resumed> )       = 0
[pid 12479] <... futex resumed> )       = 1
[pid 12478] <... futex resumed> )       = 0
[pid 12477] futex(0x76ad83c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12482] read(6,  <unfinished ...>
[pid 12481] <... read resumed> "MemTotal:         949584 kB\nMemF"...,
1024) = 1008
[pid 12479] futex(0x76ad83c0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12478] futex(0x76ad83c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12477] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12481] read(7,  <unfinished ...>
[pid 12480] futex(0x76ad83c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12482] <... read resumed> "", 1024) = 0
[pid 12481] <... read resumed> "", 1024) = 0
[pid 12482] close(6 <unfinished ...>
[pid 12481] close(7 <unfinished ...>
[pid 12480] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12482] <... close resumed> )       = 0
[pid 12481] <... close resumed> )       = 0
[pid 12482] futex(0x4841b8,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1533001602,
tv_nsec=594400705}, 0xffffffff <unfinished ...>
[pid 12480] futex(0x76ad83c0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12481] futex(0x4841b8,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1533001602,
tv_nsec=595577521}, 0xffffffff <unfinished ...>
[pid 12480] <... futex resumed> )       = 0
[pid 12479] <... futex resumed> )       = 0
[pid 12478] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12477] futex(0x76ad83c0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12480] futex(0x76ad83c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12479] sendto(3,
"\0\0\0\rllamapi2\0\0\10\0\f\26\327\357\336%\372\377\260\0\t\0\f\0\0\0"...,
1294, 0, {sa_family=AF_INET, sin_port=htons(25826),
sin_addr=inet_addr("10.0.0.1")}, 16 <unfinished ...>
[pid 12480] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12478] futex(0x76ad83c0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12477] <... futex resumed> )       = 0
[pid 12478] <... futex resumed> )       = 0
[pid 12476] futex(0x76ad83c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12477] futex(0x76ad83c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12480] futex(0x76ad83c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12479] <... sendto resumed> )      = 1294
[pid 12478] futex(0x76ad83c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12479] futex(0x76ad83c0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12478] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12483] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 12479] <... futex resumed> )       = 1
[pid 12483] futex(0x48416c, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12478] futex(0x76ad83c0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12476] <... futex resumed> )       = 0
[pid 12478] <... futex resumed> )       = 1
[pid 12477] <... futex resumed> )       = 0
[pid 12483] <... futex resumed> )       = 0
[pid 12479] futex(0x76ad83c0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12483] sysinfo( <unfinished ...>
[pid 12480] <... futex resumed> )       = 0
[pid 12483] <... sysinfo resumed> {uptime=1385325, loads=[3072, 3360,
128], totalram=972374016, freeram=89022464, sharedram=258048,
bufferram=68472832, totalswap=127922176, freeswap=127303680,
procs=138, totalhigh=0, freehigh=0, mem_unit=1}) = 0
[pid 12480] futex(0x76ad83c0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12479] <... futex resumed> )       = 1
[pid 12483] futex(0x484248, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12480] <... futex resumed> )       = 0
[pid 12483] <... futex resumed> )       = 0
[pid 12480] futex(0x4841fc, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12483] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12480] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12483] <... futex resumed> )       = 0
[pid 12480] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12483] futex(0x4841b8,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1533001602,
tv_nsec=596605016}, 0xffffffff <unfinished ...>
[pid 12480] <... futex resumed> )       = 0
[pid 12479] futex(0x4841fc, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12478] futex(0x484248, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 12480] futex(0x48424c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 12479] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12478] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12479] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12478] futex(0x4841fc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12479] <... futex resumed> )       = 0
[pid 12478] <... futex resumed> )       = 0
[pid 12479] futex(0x48424c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 12478] futex(0x48424c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 12477] futex(0x76ad83c0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 12476] futex(0x76ad83c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12477] <... futex resumed> )       = 0
[pid 12476] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 12477] futex(0x48424c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 12476] futex(0x76ad83c0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 12476] futex(0x48424c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 12475] <... nanosleep resumed> 0x7ef355e8) = 0


I see a lot of 'Resource temporarily unavailable' and 'Connection timed out'.

Any other suggestions?



More information about the collectd mailing list