[collectd] Network plugin - ensuring delivery and local caching during network outages

[Andrés J. Díaz]

Hi Spike

I'm working on this problem from a long time, the main handicap below
this feature is that collectd is agnostic about the output format, that is,
the rrdtool plugin write data in RRDs, but csv plugin does in CSV, so,
when we need to resend data to server on outage recover, which storage
we use? all of them? only the first configurated? What happened if before
the outage we use RRDs, and after CSV? It's not an easy answer.

Now I implemented a local caches in my hosts, using rrdtool plugin, and
serves the rrds using a local rrdtool server (read the rrdtool man :D) over TCP,
so in my GUI, I have a check box to get graph from the remote cache or from
the server. This is not a very cool workarround, but works fine for me.

I do not known if someone has planned to add a native feature in network
plugin...

Regards,
  Andres

2009/8/10 Spike Spiegel <fsmlab at gmail.com>:
> Hi,
>
> we're currently reviewing collectd as an alternative to ganglia for
> our infrastructure and ran into a problem with the way data is handled
> during network outages and collectd is configured to use the network
> plugin. Basically what we're after is a local cache for the duration
> of the outage so metrics can be stored on the local node and then
> replayed once the connection is re-established. Given that I'm new to
> collectd I'd like to hear from people recommendations on how to
> implement this feature, either as an option to the network plugin or
> maybe as something entirely different (output to rrd or something and
> then resend via the network plugin if any data is found?
>
> I'd also be interested to hear opinions on a tcp based implementation
> in case the user wanted delivery to be guaranteed. I appreciate the
> overhead of TCP, especially for high sampling rates, but if designed
> to be an option the user can turn on or off I believe it could be
> useful for important nodes where data loss is not acceptable.
>
> thanks for your input,
>
> Spike
>
> --
> "Behind every great man there's a great backpack" - B.
>
> _______________________________________________
> collectd mailing list
> collectd at verplant.org
> http://mailman.verplant.org/listinfo/collectd
>