[collectd] collectd uses a lot of virtual memory

[matthew sporleder]

On Fri, Aug 28, 2009 at 12:42 PM, Christoffer
Sawicki<christoffer.sawicki at gmail.com> wrote:
> Hello!
>
> On Fri, Aug 28, 2009 at 10:15, Florian Forster<octo at verplant.org> wrote:
>> Hi Christoffer,
>>
>> On Thu, Aug 27, 2009 at 10:02:52PM +0200, Christoffer Sawicki wrote:
>>> VSZ is 67952 which seems a lot to me. Is this normal? Is there anyway
>>> I can lower it (perhaps by using more lazy memory allocation)? I care
>>> about the amount of allocated memory (and not only used memory) since
>>> it is counted against my VPS quota.
>>
>> yes, this is expected and known behavior: collectd uses a lot of
>> threads, which need a (fixed size) stack each. On my Linux system the
>> default stack size for threads is 8 MBytes. Since there are at least
>> 6 threads in the default configuration, you'd end up with 48+ MBytes of
>> virtual memory. 68 MBytes doesn't seem to be far off.
>
> Aha! That explains it. I now remember that I tweaked Apache's stack
> size setting some time ago because of the same reason.
>
>> Usually this is not a problem, even vor embedded systems, because Linux
>> does overcommitting by default. If this is not an option or, as in your
>> case, virtual memory size is limited (why, by the way?),
>
> This is not my domain of expertise, but here's what I've figured:
>
> * Linux instances virtualized by OpenVZ don't have a swap. Since
> there's no swap, overcommitting would be dangerous since that could
> lead to processes allocating virtual memory that would later be
> inaccessible (since there's no real memory backing it). I'm not too
> fond of random segfaults.
>
> * OpenVZ Linux instances don't have a swap so that VPS provider
> customers need to pay for more RAM. :-)
>
>> you'll have to
>> reduce the stack size for new threads. You can do that by setting the
>> appropriate ulimit prior to starting collectd, for example in the init-
>> script or (under Debian) in /etc/default/collectd.
>
> I use daemontools so I edited my run script, for the record.
>
> I'm thinking that I perhaps should set this setting globally; collectd
> is not the only application on my system that uses threads.
>
>> The command would
>> look somewhat like this:
>>
>>  # Set soft stack limit to 512 kBytes
>>  $ ulimit -S -s 512
>>
>> I can't probmise 512 kBytes to be enough for every configuration, but I
>> think it should for in most cases. You might even be able to go lower.
>
> I've been running collectd with a stack size of 256 KB for a few hours
> now without any problems. The new VSZ (virtual memory size) is 8048
> (instead of 67952) -- much friendlier too my system and wallet! :)
>
> Thanks a lot for your answer and a great application.


Any disk space can be used as swap if you have root access to make a swap file.