[collectd] Announcing “Collection 4” (C4)

[Florian Forster]

Hi everybody,

I've been working on a new front-end for collectd for some time and
finally reached a state worth publishing. “Collection 4” (abbreviation
“C4”) is intended to supersede "Collection 3" eventually, but this
doesn't mean that I won't accept patches for Collection 3 anymore, of
course. C4 is far from finished, so expect changes in functionality
frequently and also in between patch releases.

The main goal of C4 is performance. Collection 3 had a lot of startup
overhead, because a lot of Perl-Modules needed to be parsed even when
only totally unrelated graphs were to be drawn ([0]). C4 uses FastCGI to
use the same process to handle multiple requests, reducing startup
costs several magnitudes. Because the available graphs and the list of
required files for each are kept in memory, most requests can be handled
without disk access once the process has been initialized. In my
experience, the browser is the bottleneck now. C4 tries to be friendly
in this respect by limiting the amount of data sent to the browser to a
useful size ([1]).

The second central feature is graph configuration. While many front-ends
do a great job of displaying arbitrary data nicely, ultimately I want to
be able to configure the graphs to look exactly the way I want them to
look. That's why that feature was present in Collection 3 and in
Collection 2, too. The configuration syntax used in C4 is more flexible
than the syntax used in Collection 3. It allows create a graph from
multiple identifiers (think: files) but also allows one identifier to
appear in multiple graphs. Unfortunately, this is not yet fully
implemented to users cannot see this feature in all its glory.

Another important aspect are "data providers": C4 doesn't entirely focus
on RRD files as the source of its data, but abstracts away the source of
the data to make it easy to add new "providers" in the future. This
means, however, that reading ("fetching") the data needs to be separate
from rendering ("graphing") the data ⇒ no rrdgraph(1) ([2]). Currently
graphs are rendered using gRaphaël, which currently lacks many graphing
features supported by the configuration (hence the limitation in the
previous paragraph). In the near future I hope to see a data provider
for RRDCacheD (using "rrdc_fetch") and possibly CSV files.

The HTML-based interface provided by the front-end is very simple and
leaves a lot of room for improvements. Patches are especially welcome in
this respect. (Almost?) all the actions are also available in a JSON-
encoded form. My hope is that better front-end programmers than me use
this to create nice front-ends based on C4 ([3]), basically using it as a
back-end ([4]).

So, without further ado, here's the link to the homepage, wiki page and
download links:

  Homepage:  http://octo.it/c4/
  Wiki:      http://collectd.org/wiki/index.php/Collection_4
  Download:  http://octo.it/c4/files/collection-4.0.0.tar.bz2

Needless to say: Feedback is *very* welcome. So are patches :)

Best regards,
—octo

[0] When using the front-end on a smaller scale, you probably don't
    notice this too much, but when you have 50k+ RRD files and the
    system is very busy just updating RRD files, you can deliver maybe
    two graphs per second. I would have expected loading of Perl modules
    to be a CPU-bound operation once the files containing the modules
    are in the page cache, but experience tells a different story.
[1] For example, printing a list of all hosts may result in 100+ kByte
    sent to the browser on large setups. Having such a list in the menu
    may be nice when you have 10 hosts. When you have 1000, this will
    render the interface slow.
[2] This is only a half-truth: Support for rrdgraph(1)-based images is
    currently present but not very well maintainable. It *might* be
    removed in the future or at very least it will be reorganized.
[3] Lindsay / Visage, I'm looking at you! ;)
[4] Hey, if it's a front-end and a back-end at the same time, did I just
    create one of those "middlewares" businesses always get all excited
    about?! ;)
-- 
Florian octo Forster
Hacker in training
GnuPG: 0x0C705A15
http://octo.it/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: Digital signature
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20100913/ba8b088e/attachment.pgp>