[collectd] Suggestions for implementation of seperation of collection server from display/analysis

[Florian Forster]

Hi Doug,

On Fri, Feb 06, 2009 at 05:55:21PM -0700, Douglas Straub wrote:
> The documentation seems to indicate that it is a bad idea to have
> the visual display/analysis on the same server that is performing the
> collection.

well, it's not really harmful, but if the collectd server is under
heavy load due to RRD updates, creating graphs on the same machine will
impose further load and may take some time due to high latencies when
reading files.

If your system is fine, because the RRD files are on an iSCSI share,
file server, SAN or similar, or you just don't have this amount of
files, there is no problem in creating the graphs on that machine, too.

> I was wondering if anyone has any suggestions on how to implement this
> seperation. That is, to have one server that is running the
> "collectd", and to have a second server that is responsible for
> display and analysis.

I've seen basically two versions, but others may exist:

 1) Put the RRD files on a storage. Storage systems often have enough
    RAM to keep all RRD files in cache and/or can take the load. You can
    then mount the share from both servers, possibly in read-only mode
    on the one creating graphs.

 2) Store the RRD files locally and export the files using a network
    file system, for example NFS. Then mount that share from the
    graphing / analysing server.

> The second server would pull data from the collectd server every so
> often (say every 6 hours, or every day, etc). It would do web display
> of stats, and do other analysis of the gathered data. Appreciate any
> suggestions on how to do this efficiently?

I prefer graphs that are created on the fly, because I usually need the
most recent data and because creating thousands of graphs that nobody
will look at before they are overwritten seems like such a waste of
resources. You experience may differ, of course, depending on how you
use the data.

If you're primarily interested in long term trends, you could implement
a compromise which serves a graph, checks the local RRD file's age and,
if the RRD file is too old, fetches an updated version. This way you
don't have any unnecessary copy operations, graphs generations, and if
your users usually look at the long term graph first (before diving into
the details) the updates RRD file should be available then.

Hope this helps..?

Regards,
-octo
-- 
Florian octo Forster
Hacker in training
GnuPG: 0x91523C3D
http://verplant.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mailman.verplant.org/pipermail/collectd/attachments/20090210/fd11a03a/attachment.pgp