[collectd] Limit the memory used by the network plugin

[Yves Mettier]

Hello,

When rrdcreate starts to create a lot of new rrds, it slows collectd.
This is not new : issue #75.

A side effect is that while the rrd files are created, the 
dispatch_thread() function (in network.c) is freezed (at parse_packet() 
call at the end of dispatch_thread()).

Consequence : the receive_thread() (and more exactly network_receive()) 
thread continues to receive packets from the network and store them into 
memory.

I noticed the TODO line (possible performance enhancement : do not free 
the entries in the dispatch thread but put them in another list, so we 
don't have to allocate more and more...
But I don't think this will do much if we receive more and more packets 
and the dispatch_thread do not eat them.


Fixing issue #75 solves the problem ? I think not, even if it really 
improve the situation. I think not because if too many packets are sent 
and there are not enough threads to "eat" them, the problem remains.

--> I think we should limit the max number of packets to accept (or 
limit memory : divide by sizeof(structure...)).
If we don't, the OOMkiller does it for us before we get a "network 
plugin : malloc failed".

What to do if the max number of packets to accept is reached ? Well, 
drop packets. Drop last packets, or accept them but drop the older ones 
? I don't know (need to think to that, or allow the admin to configure 
it in the config file).
Note that if you have a dedicated collectd for rrd files creation, 
dropping packets is not so important : they will come back in a few 
minutes.


Does anyone have a better solution ?
Is someone already working on this ?
May I try something (and submit a patch) ?

Regards,
Yves
-- 
- Homepage       - http://ymettier.free.fr                             
-
- GPG key        - http://ymettier.free.fr/gpg.txt                     
-
- C en action    - http://ymettier.free.fr/livres/C_en_action_ed2.html 
-
- Guide Survie C - http://www.pearson.fr/livre/?GCOI=27440100673730    
-