[collectd] Shouldnt collectd also provide an interface to the data?

[Lindsay Holmwood]

G'day,

2009/9/13 Spike Spiegel <fsmlab at gmail.com>:
>
> The main problem I see is that new frontends and GUIs come up tied to
> some application the developer is using and while that's normal I
> think it doesn't help the cause as much as it could. Effectively what
> is being discussed here is a generic toolkit to interact with/graph
> data stored by a monitoring system, and also focusing exclusively on
> rrd is possibly limiting.
>

By "application", do you mean the statistics collection app like
collectd or Ganglia, or the data abstraction like Visage?

I'm all for building a generic solution to the world's graphing
problems, but for the time being I think the focus should be on
improving the graphs of individual statistic collection apps, then
genericise later. There are enough inconsistencies in the collectd
data to keep me busy for a while. :-)

People will have strong opinions on how they want their data graphed.
Some will think JS is insane, while others will think that server side
rendered images are too brittle.

Where the graphs are rendered will strongly influence how the reusable
the code is. Client side code won't be usable on the server side, and
vice versa.

> On Sat, Sep 5, 2009 at 7:26 AM, Mariusz Gronczewski <xani666 at gmail.com> wrote:
>> 2009/9/4 Lindsay Holmwood <lindsay at holmwood.id.au>:
>>> 2009/9/4 Mariusz Gronczewski <xani666 at gmail.com>:
>>>> There is collection which is nice
>>>> for start, drraw, which is good but its a bit ugly and afaik dont
>>>> don't have some advanced RRDtool functions implemented and thats it.
>>>> If u want something like "get graph with stacked all loadavg's from
>>>> all servers" or "get total network bandwidth from all servers" u have
>>>> to write a script which generates it, or update graph in drraw
>>>> everytime u add/delete host.
>
> true, which is what multiple tools have attempted to do with several
> degree of success, but again tied to a specific application. Wouldn't
> it be better to have a generic graphing UI that implemented some API
> to retrieve  metrics for  nodes in a given time range?
>

I'm part way there with Visage right now. The JavaScript that
generates the graphs (visageGraph) is abstracted away from the fetch
phase (visageBase). I've done it this way so I can easily build
different graph types while utilising the same fetch code.

Right now the fetch code is tied to the the Visage JSON api, but it
wouldn't take a lot to refactor if someone wants to take up that
mantle.

>>>
>>> The other problem with RRDs is that they generally require you to have
>>> access to the files themselves to do anything useful with them.
>
> indeed, so here something has to be done, either by writing a web
> service like visage or maybe using rrdcached, point again is, it's
> irrelevant to the UI and graphs generation

Actually it's pretty relevant, if you don't want god-awful
normalisation in your graphing code.

I've spent the last week battling inconsistencies in the way collectd
stores its RRD data. collectd uses 2 different conventions:

plugin/plugin_instance => datasource, e.g.
cpu-0/cpu-load => value
load/load => shortterm, midterm, longterm

Even within that plugins log data slightly different (e.g. multiple
plugin instances relating to a single plugin, like the cpu plugin),
and the processing phase gets tricky, regardless of whether the code
is client or server side.

I don't know what form Ganglia's RRDs are, but I assume there are some
corner cases that will make it difficult.

>
>> Yeah but then for keeping it in some kind of db
>
> I lost you here, sorry, why would you want to keep the data in some kind of db?

If you don't want large RRDs containing lots of historical data.

>>
>> And if ure interested only in actual data u can run collectd on
>> "graph" node and get data using unixsock plugin
>
> what about historical data? the graph node shouldn't be active 24/7,
> graphs should be on demand. Am I missing something here?

See above.

>
> Another couple bits I wanted to mention:
> 1) RRD while being a very common solution is not the only one, writing
> a fantastic UI that can make stacked graphs etc and not be able to
> feed it data from another source is an issue to keep in mind

As long as the data fetching phase is loosely coupled to the graph
generation phase it won't be an issue. It's all about time series
data, right? As long as you're thinking about graphing time series
data (and not a specific application's data) you'll mitigate most
future problems.

> 3) An eye should be kept on metadata. One of the problems of just
> reading a rrd file is that it doesn't tell you anything about the host
> it belongs to, its location, duty and so on. Some applications have
> that kind of metadata attached so imho it'd be useful to design a UI
> so that if provided it could fetch metadata from some source.

Where that data is stored usually gives you that information
(basename, filename). I totally agree that data should be exposed via
the same fetch service.

Opinions? Lets keep the discussion going!

Cheers,
Lindsay


-- 
http://holmwood.id.au/~lindsay/ (me)