[collectd] TCP support in network plugin

[Konstantin Tokarev]

12.06.15, 09:53, "Andreas Maus" <a.maus at science-computing.de>:

>Hi *.
>
>On Do Jun 11, 2015 at 23:24:59 +0300, Konstantin Tokarev wrote:
>> 
>> 
>> 11.06.2015, 21:56, 'Florian Forster' <octo at collectd.org>:
>> > Hi Konstantin,
>> >
>> > On Thu, Jun 11, 2015 at 03:08:04PM +0300, Konstantin Tokarev wrote:
>> >>  Is it a design desition that network plugin supports UDP and
>> >>  multicast, but not TCP?
>> >
>> > yes, it allowed us to implement support for multicast, which is not
>> > possible with TCP.
>> 
>> Whats wrong with just sending the same packets over TCP connection?
>Connection overhead and traffic overhead. Especially for large multicast
>groups the traffic reduction (compared to n single TCP connections) is
>obvious.
>
>The approach of collectd make it a very suitable solution for collecting
>LOTS of metrics for lots of systems (I collect about about 250k metrics
>every 10 sec. for about 1000 hosts and Ive seen larger setups).
>Multicast or not multicast aside, the connection overhead for large
>number of TCP connections is much higher for the server compared to UDP connection
>handling. Including hammering the servers with lots of TCP handshakes
>after a planned maintenance or other problems with lot of half open connections
>(Not including additional 'hidden' resource requirements like memory
>consumption on routers or firewall devices for the state tables).
>
>So TL;DR: I think its a design decision (and not a bad one ;) )

Note that I've never meant that TCP should be a default option.

>
>> >
>> >>  I think using TCP would be better option than UDP for transmitting
>> >>  data over Internet. Otherwise, collectd would need to mitigate UDP
>> >>  packet loss somehow.
>The key here is 'Internet' am I right?

Exactly.

>Using a TCP connection to transmit data over a 'noisy' line or a 
>WAN link is indeed a better solution to prevent loss of packets
>or other problems like reordering of UDP packets
>(and dont get me started on ISPs with broken configuration
>for handling fragmentation and reassembly of large UDP packets;) )
>
>Using UDP over WAN links may be problematic. In this case
>you should consider using AMQP as suggested or a TCP based
>tunnel between your endpoints. 
>
>> >
>> > You can work around this with the AMQP plugin, for example. Id also
>> > like to have a reliable-by-default option, but the idea I have in mind
>> > is not quite there yet …
>> Im planning to use collectd on embedded systems, and AMQP requires
>> additional dependency. Also, its not a binary protocol.
>*errr* The protocol itself is binary (see http://www.amqp.org/resources/download,
>especially the section 'AMQP Wire-Level Format' of the protocol
>specification).

Sorry, I've misunderstood documentation. Yes, AMQP is a binary protocol, but collectd payload is sent in a text format. Binary part of packet is actually irrelevant for data transfer between two collectd instances when there are no other publishers and subscribers.


>
>Depending on the setup of your embedded systems (and your configuration
>setup) the resource requirements for a AMQP producer isnt that much.
>But YMMV ;)

I will try it, but whole publish-subscribe thing looks like an overkill to me. Btw, I'm concerned not only about resource usage, but also about firmware binary size and simplicity of build process.
>
>My 2 euro cents,
>
>Andreas.
>
>-- 
>Dipl.-Ing. Andreas Maus             science+computing ag
>System Administration               Hagellocher Weg 73
>tel.: +49 7071 9457 671             72070 Tuebingen, Germany
>fax: +49 7071 9457 411              www.science-computing.de
-- 
-- <br />Regards,
Konstantin