[collectd] Shouldnt collectd also provide an interface to the data?

[Mariusz Gronczewski]

Hi,

>> On Sat, Sep 5, 2009 at 7:26 AM, Mariusz Gronczewski <xani666 at gmail.com> wrote:
>>> 2009/9/4 Lindsay Holmwood <lindsay at holmwood.id.au>:
>>>> 2009/9/4 Mariusz Gronczewski <xani666 at gmail.com>:
>>>>> There is collection which is nice
>>>>> for start, drraw, which is good but its a bit ugly and afaik dont
>>>>> don't have some advanced RRDtool functions implemented and thats it.
>>>>> If u want something like "get graph with stacked all loadavg's from
>>>>> all servers" or "get total network bandwidth from all servers" u have
>>>>> to write a script which generates it, or update graph in drraw
>>>>> everytime u add/delete host.
>>
>> true, which is what multiple tools have attempted to do with several
>> degree of success, but again tied to a specific application. Wouldn't
>> it be better to have a generic graphing UI that implemented some API
>> to retrieve  metrics for  nodes in a given time range?
>>
>
> I'm part way there with Visage right now. The JavaScript that
> generates the graphs (visageGraph) is abstracted away from the fetch
> phase (visageBase). I've done it this way so I can easily build
> different graph types while utilising the same fetch code.
>
> Right now the fetch code is tied to the the Visage JSON api, but it
> wouldn't take a lot to refactor if someone wants to take up that
> mantle.
>
>>>>
>>>> The other problem with RRDs is that they generally require you to have
>>>> access to the files themselves to do anything useful with them.
>>
>> indeed, so here something has to be done, either by writing a web
>> service like visage or maybe using rrdcached, point again is, it's
>> irrelevant to the UI and graphs generation
>
> Actually it's pretty relevant, if you don't want god-awful
> normalisation in your graphing code.
>
> I've spent the last week battling inconsistencies in the way collectd
> stores its RRD data. collectd uses 2 different conventions:
>
> plugin/plugin_instance => datasource, e.g.
> cpu-0/cpu-load => value
> load/load => shortterm, midterm, longterm
>
> Even within that plugins log data slightly different (e.g. multiple
> plugin instances relating to a single plugin, like the cpu plugin),
> and the processing phase gets tricky, regardless of whether the code
> is client or server side.
>
> I don't know what form Ganglia's RRDs are, but I assume there are some
> corner cases that will make it difficult.
>
About getting access to data, you could possibly just give access to
rrd files thru normal HTTP server and then just add in graphing
software possibility to remotely fetch RRD/CSV/whatever from webserver
>>
>>> Yeah but then for keeping it in some kind of db
>>
>> I lost you here, sorry, why would you want to keep the data in some kind of db?
>
> If you don't want large RRDs containing lots of historical data.
If u want historic data u have to have some kind of DB, RRD is just a
form of it. If u want to store data in for example standard SQL DB ull
have to write some code to clean/merge historic data (like RRD does,
older data points gets "averaged" together) and ull need much more I/O
power for same amount of monitored nodes
>>>
>>> And if ure interested only in actual data u can run collectd on
>>> "graph" node and get data using unixsock plugin
>>
>> what about historical data? the graph node shouldn't be active 24/7,
>> graphs should be on demand. Am I missing something here?
>
> See above.
Graphing software could fetch historic data from rrd/csv/whatever and
then just use unixsock to get actual values for realtime graphs ;]

>>
>> Another couple bits I wanted to mention:
>> 1) RRD while being a very common solution is not the only one, writing
>> a fantastic UI that can make stacked graphs etc and not be able to
>> feed it data from another source is an issue to keep in mind
>
> As long as the data fetching phase is loosely coupled to the graph
> generation phase it won't be an issue. It's all about time series
> data, right? As long as you're thinking about graphing time series
> data (and not a specific application's data) you'll mitigate most
> future problems.
Basically, software like cacti or drraw use RRDtool as graphing engine
and thats why they are tied to RRD. Usually problem with graphing
software is that its easy to make simple graphs, but complicated (or
impossible) to make slightly more advanced ones. For example making
mathematical operation on datasources before turning it into graph in
RRD is fairly simple (
http://oss.oetiker.ch/rrdtool/doc/rrdgraph_rpn.en.html ) once u get
RPN notation (wish they used something more "friendly"). Also
calculating things like "total bytes transferred" or calcualting 95th
percentile.

Regards
Mariusz