[collectd] Collection station for interface performance data, storing the data in a MySQL db

[Bart van den Heuvel]

Thanks for this insight.

yes, It has been on my wishlist for a while. The problem with storing data
in rrd files is IO. With lots of interfaces there's lots of overhead. While
it's true that a RDMS solution needs maintenance, maybe even hourly, it does
make things possible that would be very hard in rrd solutions.

Top N interfaces, Top N errors, etc. I might start a project to test the
concept, see where it brings us. Maybe a lightweight poller based on C that
gets target configdata (ip and communitystring) from a RDMS and places the
gathered data back in the db. Data is then handled like rrd, simplefying the
data with maintenance jobs based on time.

I don't know if i would prefer the/a plugin for collectd. Collectd seems a
100.000 collectors toolbox, i only need the snmp/mysql collection and
storing. If i was looking for a collect the world agent for servers and
networkcomponents this would be the best!
It also seems that the setup for collectd is modular, wich would enable me
to keep it clean.

Also the thought of doing it al in a single config file for thousands of
interfaces seems a bit errorprone.

Thanks! Regards, Bart

2009/9/25 Florian Forster <octo at verplant.org>

> Hi Bart,
>
> On Tue, Sep 22, 2009 at 11:41:29AM +0200, Bart van den Heuvel wrote:
> > Can Collectd fetch data from lots of interfaces and then store it into
> > a MySQL db?
>
> no, currently collectd cannot store data *in* MySQL or any other
> relational database system (RDBMS).
>
> The problem is the databases: If you have only 100 new values per
> second, this translates to 8,640,000 new values every day. This becomes
> a problem for databases pretty quickly, so you need to be clever there
> and partition the data.
>
> There's a sample PostgreSQL script [0] by Bob Cotton, which uses the
> “Star schema” [1] to partition the data. The idea is basically to have
> one table to hold host/plugin/type and separate tables for the actual
> values (those are the ones growing fast). For the value tables you
> create one table for each day (or every 6 hours or whatever is
> reasonable for your setup).
>
> What keeps me from including Bob's SQL script together with a C based
> plugin to write data into the PostgreSQL database is that the SQL script
> currently needs a maintenance job to run once a day. I'd much prefer if
> the SQL script created required tables as it needs them. Unfortunately,
> my SQL-foo is not good enough to adapt that.
>
> The matter has been on your wishlist [2] for a while now. If you (or
> anyone else on the list) is interested to work into that direction,
> please let me know :)
>
> Regards,
> —octo
>
> [0] <http://github.com/bcotton/collectd_dbstore>
> [1] <http://en.wikipedia.org/wiki/Star_schema>
> [2] <http://collectd.org/wiki/index.php/Roadmap#Store_to_RDBMS>
> --
> Florian octo Forster
> Hacker in training
> GnuPG: 0x91523C3D
> http://verplant.org/
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFKvGzQHdggu3Q05IYRAmcQAKCPuE7TrMJxB3aaz8q6tpiOVwtqdgCfeh7A
> ⛧꽗鴽凞妉핰◪�=
> =RVBt
> -----END PGP SIGNATURE-----
>
>


-- 
Groeten,

Bart van den Heuvel

Any society that would give up a little liberty to gain a little security
will deserve neither and lose both.
Benjamin Franklin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.verplant.org/pipermail/collectd/attachments/20090925/269ef896/attachment.htm