Thanks for this insight.<br><br>yes, It has been on my wishlist for a while. The problem with storing data in rrd files is IO. With lots of interfaces there's lots of overhead. While it's true that a RDMS solution needs maintenance, maybe even hourly, it does make things possible that would be very hard in rrd solutions.<br>
<br>Top N interfaces, Top N errors, etc. I might start a project to test the concept, see where it brings us. Maybe a lightweight poller based on C that gets target configdata (ip and communitystring) from a RDMS and places the gathered data back in the db. Data is then handled like rrd, simplefying the data with maintenance jobs based on time.<br>
<br>I don't know if i would prefer the/a plugin for collectd. Collectd seems a 100.000 collectors toolbox, i only need the snmp/mysql collection and storing. If i was looking for a collect the world agent for servers and networkcomponents this would be the best!<br>
It also seems that the setup for collectd is modular, wich would enable me to keep it clean. <br><br>Also the thought of doing it al in a single config file for thousands of interfaces seems a bit errorprone.<br><br>Thanks! Regards, Bart<br>
<br><div class="gmail_quote">2009/9/25 Florian Forster <span dir="ltr"><<a href="mailto:octo@verplant.org">octo@verplant.org</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi Bart,<br>
<div class="im"><br>
On Tue, Sep 22, 2009 at 11:41:29AM +0200, Bart van den Heuvel wrote:<br>
> Can Collectd fetch data from lots of interfaces and then store it into<br>
> a MySQL db?<br>
<br>
</div>no, currently collectd cannot store data *in* MySQL or any other<br>
relational database system (RDBMS).<br>
<br>
The problem is the databases: If you have only 100 new values per<br>
second, this translates to 8,640,000 new values every day. This becomes<br>
a problem for databases pretty quickly, so you need to be clever there<br>
and partition the data.<br>
<br>
There's a sample PostgreSQL script [0] by Bob Cotton, which uses the<br>
“Star schema” [1] to partition the data. The idea is basically to have<br>
one table to hold host/plugin/type and separate tables for the actual<br>
values (those are the ones growing fast). For the value tables you<br>
create one table for each day (or every 6 hours or whatever is<br>
reasonable for your setup).<br>
<br>
What keeps me from including Bob's SQL script together with a C based<br>
plugin to write data into the PostgreSQL database is that the SQL script<br>
currently needs a maintenance job to run once a day. I'd much prefer if<br>
the SQL script created required tables as it needs them. Unfortunately,<br>
my SQL-foo is not good enough to adapt that.<br>
<br>
The matter has been on your wishlist [2] for a while now. If you (or<br>
anyone else on the list) is interested to work into that direction,<br>
please let me know :)<br>
<br>
Regards,<br>
—octo<br>
<br>
[0] <<a href="http://github.com/bcotton/collectd_dbstore" target="_blank">http://github.com/bcotton/collectd_dbstore</a>><br>
[1] <<a href="http://en.wikipedia.org/wiki/Star_schema" target="_blank">http://en.wikipedia.org/wiki/Star_schema</a>><br>
[2] <<a href="http://collectd.org/wiki/index.php/Roadmap#Store_to_RDBMS" target="_blank">http://collectd.org/wiki/index.php/Roadmap#Store_to_RDBMS</a>><br>
<font color="#888888">--<br>
Florian octo Forster<br>
Hacker in training<br>
GnuPG: 0x91523C3D<br>
<a href="http://verplant.org/" target="_blank">http://verplant.org/</a><br>
</font><br>-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v1.4.6 (GNU/Linux)<br>
<br>
iD8DBQFKvGzQHdggu3Q05IYRAmcQAKCPuE7TrMJxB3aaz8q6tpiOVwtqdgCfeh7A<br>
⛧꽗鴽凞妉핰◪�=<br>
=RVBt<br>
-----END PGP SIGNATURE-----<br>
<br></blockquote></div><br><br clear="all"><br>-- <br>Groeten,<br><br>Bart van den Heuvel<br><br>Any society that would give up a little liberty to gain a little security <br>will deserve neither and lose both. <br>Benjamin Franklin<br>
<br>