[collectd] I guess we need a config file

[Niki Waibel]

> right, CPU performance is not an issue. The time the kernel or daemon or
> whoever needs to get a lock on some counter is more likely to be a
> bottleneck, but I'm positive that's not a problem at all.
> 
> The second slowest operation collectd does is opening files. The
> operating system is quite good at opening files, so that's not the
> problem either.
> 
> So one and only problem so far is sockets. So far the modules using
> sockets are `ping' and `hddtemp'. `hddtemp' connects to localhost which
> should be reasonably fast. `ping' on the other hand waits for the other
> host to answer. If we assume an average roundtrip time of 20ms collectd
> simply sits there and waits for 172.8 seconds every day. This value
> (linearly) grows with the number of hosts you ping. This is my
> motivation to look into threads.

okay, so main motivation is not to parallelize the main loop.
it is about making some things in some modules parallel.
(maybe the main loop will be parallel later... ;-)

well, if you start all ping threads with pth, they'll wait via
read for an answer. this is almost the ideal thing for pth.

example:
main thread
        \------ping sent to host1
        \------ping sent to host2
        \------ping sent to host3
all 3 processes are waiting.
1 of them (let's say host2 answers) first.
then the function will finish *without* any other things in parallel.
maybe within that time host3 had answered also. then that thread will be
finished as well, but (non-preemptive) !AFTER! thread host 2 was finished.
then, maybe 5 sec later host1 answers and the last thread can be finished.

so instead of
        ping host1, wait, host1 answers, process stats for host1, (ie takes 5 sec)
        ping host2, wait, host2 answers, process stats for host2, (ie takes 1 sec)
        ping host3, wait, host3 answers, process stats for host3, (ie takes 1 sec)
(7sec) it would be like (5sec)
        ping host1, wait, ping host2, wait, ping host3, wait, (takes 1sec then host2 and 3 answered)
        process stats for host2, (i think this takes less then 0.1 sec)
        process stats for host3, (i think this takes less then 0.1 sec)
        (now, after 4 more sec, host 1 answers)
        process stats for host1
the time diff is not big in this example. but imagine that
there are 100 hosts, 20 answer in 5sec, 80 in 1sec:
you'd now need at least 20*5sec + 80*1sec = 180sec (without threads).
using pth you'd need (ideally) a bit more then 5 sec. in reality
maybe 6-7 sec (processing of the answers).

using preemptive threads things could be finished within maybe
about 5.1sec.

so, again, i see no need to introduce complicated preemptive
task switching.

>> the advantage (in regard to collectd, imho) of the gnu pth is
>> that they are non-preemptive. another thread runs only
>>         a) if the program requests it
>>         b) if the running thread starts waiting for sthg
>> this has the advantage that the modules need not take care of any
>> concurrency. basically you can just leave them as they are!!! no
>> locking of curtime or any global var is necessary.
> 
> Are you sure this will result in a speedup? If I have only one
> kernel-thread, doesn't that mean that the entire programm has to wait
> for a syscall sooner or later?

as you see above (thanks a lot that you've mentioned the ping module!!!)
you def have a speed improvement, also with non-preemprive threads.

> Although I am thinking about preemptive, parallel threads I don't think
> the plugins need much attention: What I want is one thread for each
> plugin, so the functions within a plugin are not called in parallel.
> Places where locks may be neccessary would be
> - plugin_submit
> - rrd_update_file
> - plugin_* (the routines that read/change the linked list)
> - cf_*     (the routines that read/change the linked list)
> 
> We can ignore the `plugin_*' and `cf_*' functions if we do
> initialization sequencially..

:) well, usually there is always sthg which is not seen
in the first place. with pth you can just be sure that
you'll not run into troubles.

>> also some system functions are not preemptive thread save,
>> but most of them are non-preemptive thread save.
> 
> This is true for system functions that allocate memory, e.g. `stat' (or
> `lstat'). So I think most functions that are used often should be okay..
> 
> So, my points are:
> - Locking issues are not that difficult to handle
> - Advantages, as I see them so far, don't justify the integration of
>   another library

well, i've warned you :)

> Of course, as long as I haven't started developing in one direction or
> another I'm still easy to convince I'm wrong ;)

not trying to convince sthg, because i've not used pth so far.
but i was programming other preemptive threads. i just remember
that it was a pain...

as i've read about the gnu pth i was just very happy that
someone realized that most of the time non-preemptive threads
are enough.

again, not convincing, just informing that pth is there as well.

rds, niki