[collectd] TCP support in network plugin

[Andreas Maus]

Hi *.

On Do Jun 11, 2015 at 23:24:59 +0300, Konstantin Tokarev wrote:
> 
> 
> 11.06.2015, 21:56, "Florian Forster" <octo at collectd.org>:
> > Hi Konstantin,
> >
> > On Thu, Jun 11, 2015 at 03:08:04PM +0300, Konstantin Tokarev wrote:
> >>  Is it a design desition that network plugin supports UDP and
> >>  multicast, but not TCP?
> >
> > yes, it allowed us to implement support for multicast, which is not
> > possible with TCP.
> 
> What's wrong with just sending the same packets over TCP connection?
Connection overhead and traffic overhead. Especially for large multicast
groups the traffic reduction (compared to n single TCP connections) is
obvious.

The approach of collectd make it a very suitable solution for collecting
LOTS of metrics for lots of systems (I collect about about 250k metrics
every 10 sec. for about 1000 hosts and I've seen larger setups).
Multicast or not multicast aside, the connection overhead for large
number of TCP connections is much higher for the server compared to UDP connection
handling. Including hammering the servers with lots of TCP handshakes
after a planned maintenance or other problems with lot of half open connections
(Not including additional "hidden" resource requirements like memory
consumption on routers or firewall devices for the state tables).

So TL;DR: I think it's a design decision (and not a bad one ;) )

> >
> >>  I think using TCP would be better option than UDP for transmitting
> >>  data over Internet. Otherwise, collectd would need to mitigate UDP
> >>  packet loss somehow.
The key here is "Internet" am I right?
Using a TCP connection to transmit data over a "noisy" line or a 
WAN link is indeed a better solution to prevent loss of packets
or other problems like reordering of UDP packets
(and don't get me started on ISPs with broken configuration
for handling fragmentation and reassembly of large UDP packets;) )

Using UDP over WAN links may be problematic. In this case
you should consider using AMQP as suggested or a TCP based
tunnel between your endpoints. 

> >
> > You can work around this with the AMQP plugin, for example. I'd also
> > like to have a reliable-by-default option, but the idea I have in mind
> > is not quite there yet …
> I'm planning to use collectd on embedded systems, and AMQP requires
> additional dependency. Also, it's not a binary protocol.
*errr* The protocol itself is binary (see http://www.amqp.org/resources/download,
especially the section "AMQP Wire-Level Format" of the protocol
specification).

Depending on the setup of your embedded systems (and your configuration
setup) the resource requirements for a AMQP producer isn't that much.
But YMMV ;)

My 2 euro cents,

Andreas.

-- 
Dipl.-Ing. Andreas Maus             science+computing ag
System Administration               Hagellocher Weg 73
tel.: +49 7071 9457 671             72070 Tuebingen, Germany
fax: +49 7071 9457 411              www.science-computing.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20150612/74ac82cd/attachment-0001.sig>