[collectd] Preventing collectd high memory usage

[Wilfried Goesgens]

Hi everyone,   

We also found out, that replacing the system allocator for collect with
jemalloc or tcmalloc significantly improves the enduring memory footprint.  


We suffer from a problem called 'heap fragmentation' here to wich the glibc's
allocation algorith is vulnerable.  

It will reserve chunks of memory from the system; and rent it to parts of the
process. Once such a block is full, a new blog will be reserved.  

However, only the very latest block will be used to rent out memory to the
application. So, if you have a single byte valid allocated for a longer time
in such a memory block, it will remain 'forever yours'.  
>  Mon Dec 14 2015 14:09:35 EST from "Vaughn Wernersbach"
><Vaughn.Wernersbach at bidstrading.com>  Subject: Re: [collectd] Preventing
>collectd high memory usage
>
>        
>      
>
>Hi André Luiz da Silva!  Thanks for the suggestion.  I’ll keep
>switching to UDP in mind, though I’m inclined to stick with TCP if at all
>possible (largely to minimize the required changes to my environment).  
>
>   
>
>All - Anyone have info on using CollectInternalStats?  Still need to know
>what value to set WriteQueueLimitHigh to and having collectd tell me
>definitely seems like the best bet.  
>
>   
>
>   
>
>   
>
>*Vaughn Wernersbach*  
>
>*BIDS Trading, L.P.*  
>
>   
>
>111 Broadway, Suite 1603 | New York, NY 10006  
>
>T: 212.618.2071 |  F: 212.618.2031  |  C: 732.397.8344  | E:
>vaughn.wernersbach at bidstrading.com  
>
>   
>
>*From:* André Luiz da Silva [mailto:andresilvaluiz1 at gmail.com] 
> *Sent:* Monday, December 14, 2015 11:18 AM
> *To:* Vaughn Wernersbach <Vaughn.Wernersbach at bidstrading.com>
> *Cc:* collectd at verplant.org
> *Subject:* Re: [collectd] Preventing collectd high memory usage  
>
>     
>
>Hi, Wenersbach! Did you try to send your metrics over UDP instead of TCP?
>It's just a suggestion if you have a reliable network.  
>
>   
>
>Regards.  
>
>   
>
>Em 14/12/2015 1:54 PM, "Vaughn Wernersbach"
><Vaughn.Wernersbach at bidstrading.com> escreveu:    
>>      
>>
>>Hi,  
>>
>>   
>>
>>In order to limit high memory usage (one of my servers hit 94% for
>>collectd) I’m interested in enabling the  WriteQueueLimitHigh option, but I
>>don’t know what value to use.  The docs suggest enabling 
>>CollectInternalStats, and that the “write_queue” plugin instance reports
>>the info from CollectInternalStats that I’m looking for.  However, I
>>don’t see any “write_queue” plugin nor do I see anything relevant being
>>written by my existing write plugins (syslog and graphite) after enabling
>>CollectInternalStats.  
>>
>>   
>>
>>·         How do I obtain data written collected when setting
>>CollectInternalStats to true?  
>>
>>·         Alternately, are there any guidelines for setting the
>>values of WriteQueueLimitHigh?  
>>
>>   
>>
>>   
>>
>>   
>>
>>*Vaughn Wernersbach*  
>>
>>*BIDS Trading, L.P.*  
>>
>>   
>>
>>111 Broadway, Suite 1603 | New York, NY 10006  
>>
>>T: 212.618.2071 |  F: 212.618.2031  |  C: 732.397.8344  | E:
>>vaughn.wernersbach at bidstradingcom  
>>
>>   
>>
>>  
>>
>>  
>>
>>
>> _______________________________________________
>> collectd mailing list
>> collectd at verplant.org
>> http://mailman.verplant.org/listinfo/collectd  

>  
>
>  
>
>  
>
>        
>  
>
>   (, 0 bytes) [View| Download]
>   
>
>  
>
>  
>
>  

  

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20151214/bc5b429f/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20151214/bc5b429f/attachment.ksh>