[collectd] Shouldnt collectd also provide an interface to the data?

[Spike Spiegel]

Hi,

there seems to be quite a bit of momentum on this front (the same
discussion is happening in ganglia world and REST APIs have been also
brought up on the rrd ML) and I think we (the monitoring community -
not solely the collectd one) should try to size the chance to build
better tools.

The main problem I see is that new frontends and GUIs come up tied to
some application the developer is using and while that's normal I
think it doesn't help the cause as much as it could. Effectively what
is being discussed here is a generic toolkit to interact with/graph
data stored by a monitoring system, and also focusing exclusively on
rrd is possibly limiting.

On Sat, Sep 5, 2009 at 7:26 AM, Mariusz Gronczewski <xani666 at gmail.com> wrote:
> 2009/9/4 Lindsay Holmwood <lindsay at holmwood.id.au>:
>> 2009/9/4 Mariusz Gronczewski <xani666 at gmail.com>:
>>>
>>> I also agree collectd is bad place for implementing any kind of
>>> frontend it would just bloat codebase. General problem with any tool
>>> using RRD is that its great piece of software but there aren't any
>>> good, easy to use "graph creators".

I don't think the issue is "graph creators" but rather "data access".
There are some decent tools to interact with data expressed in time
series, the problem is to get access to that data (and the metadata,
more below).

>>> There is collection which is nice
>>> for start, drraw, which is good but its a bit ugly and afaik dont
>>> don't have some advanced RRDtool functions implemented and thats it.
>>> If u want something like "get graph with stacked all loadavg's from
>>> all servers" or "get total network bandwidth from all servers" u have
>>> to write a script which generates it, or update graph in drraw
>>> everytime u add/delete host.

true, which is what multiple tools have attempted to do with several
degree of success, but again tied to a specific application. Wouldn't
it be better to have a generic graphing UI that implemented some API
to retrieve  metrics for  nodes in a given time range?

>>
>> The other problem with RRDs is that they generally require you to have
>> access to the files themselves to do anything useful with them.

indeed, so here something has to be done, either by writing a web
service like visage or maybe using rrdcached, point again is, it's
irrelevant to the UI and graphs generation

>> When
>> you consider a reasonably loaded collectd server pulling in lots of
>> stats, adding graphing on top of that is just going to slow you down.

this is a valid concern, but not really specific to collectd or rrd,
any storage system will present this issue and there are known ways to
deal with it (shard per node, take a snapshot and send your read
queries there, etc)

> Yeah but then for keeping it in some kind of db

I lost you here, sorry, why would you want to keep the data in some kind of db?

> Maybe answer to "you have to have access to local rrd data" would be
> some kind of daemon which does "rrdtool dump filename" over net, then
> graphing tool connected to it.

exactly
> But then you could just share it via NFS.

not really, no. for large amounts of data that you should be able to
access even from your laptop NFS is not the answer, a web service is a
much better approach (lots of freebies, easy to secure with ssl,
loadbalance etc).

>
> And if ure interested only in actual data u can run collectd on
> "graph" node and get data using unixsock plugin

what about historical data? the graph node shouldn't be active 24/7,
graphs should be on demand. Am I missing something here?

Another couple bits I wanted to mention:
1) RRD while being a very common solution is not the only one, writing
a fantastic UI that can make stacked graphs etc and not be able to
feed it data from another source is an issue to keep in mind
2) Speaking of UIs, have you seen
https://labs.omniti.com/trac/reconnoiter 's UI? if you take a look at
the video the UI is pretty good and definitely a step ahead of what
other products do
3) An eye should be kept on metadata. One of the problems of just
reading a rrd file is that it doesn't tell you anything about the host
it belongs to, its location, duty and so on. Some applications have
that kind of metadata attached so imho it'd be useful to design a UI
so that if provided it could fetch metadata from some source.

glad to see this discussion (and visage) going.

Spike

-- 
"Behind every great man there's a great backpack" - B.