[collectd] New plugin: "Aggregation" creates aggregated metrics.
Yves Mettier
ymettier at free.fr
Tue Mar 12 15:30:32 CET 2013
Hello,
I'm taking this old mail because I was trying to see how I could move
from my "basic_aggregator" plugin to the "official" "aggregation"
plugin.
I have some remarks about "aggregation" :
1/ You cannot add more than one "Host" line per aggregation. In fact,
you can, but only the last one will be taken into account. So my example
in my previous mail is wrong.
2/ If you want to add more than one host, you need a regular
expression. This is supported. However, there is a limit of size for the
regex : DATA_MAX_NAME_LEN that seems to be defined to 64 in plugin.h.
This is very short for a list of hosts. And not the best way to specify
a list of hosts (tree or hash table is better for that).
3/ If your type has more than one datasource (like "load"), aggregation
is currently not supported.
Now, I need to aggregate the same plugin-*/type-* for lists of hosts
(for example to know how much memory total is used on a computing farm,
or its total load).
So "aggregation" will not fit because it does not support lists of
hosts (or lists of instance...)
--> Will "aggregation" support lists of hosts (lists of regex too) in
the future ?
--> Because hosts lists are dynamic, reloading the aggregation config
file without restarting collectd is necessary. How can this be done ?
--> How about having 2 aggregators ? "aggregation" as "fast_aggregator"
or "light_aggregator" for example, and "basic_aggregator" renamed as
"hosts_aggregator" ? (whatever the name in fact, it's only for
illustration)
Regards,
Yves
Le 2012-11-12 10:13, Yves Mettier a écrit :
> Hello,
>
> This is a great news !
>
> As the author of a similar plugin, basic_aggregator (see my Pull
> Request #136), I'm interested in yours.
>
> Here are some of my needs and how I would use your aggregator. If
> this is correct, feel free to copy/paste to the documentation.
> But I also have questions. Please read below.
>
> Aggregation of cpu-total/cpu-idle for host1-host4
> -------------------------------------------------
> (in reality, there may be hundreds of hosts)
>
> <Plugin "aggregation">
> <Aggregation>
> Host host1
> Host host2
> Host host3
> Host host4
> Plugin "cpu"
> PluginInstance "cpu-total"
> Type "cpu"
> TypeInstance "idle"
>
> GroupBy "Host"
>
> CalculateSum true
> CalculateAverage true
> </Aggregation>
> </Plugin>
>
> The aggregator types will be :
>
> global/aggregation-cpu-cpu-total-sum/cpu-idle
> global/aggregation-cpu-cpu-total-average/cpu-idle
>
> Is that correct ?
>
> Aggregation of each of interface-*/if_octets for host1-host4
> ------------------------------------------------------------
>
> <Aggregation>
> Host host1
> Host host2
> Host host3
> Host host4
> Plugin "interface"
> Type "if_octets"
>
> GroupBy "Host"
> GroupBy "PluginInstance"
>
> CalculateSum true
> CalculateAverage true
> </Aggregation>
>
> The aggregator types will be :
>
> global/aggregation-interface-sum/if_octets
> global/aggregation-interface-average/if_octets
>
> Is that correct ?
>
> And what if I want to aggregate them by host only ? Do I have to do a
> script that get the list of all pluginInstance (collectdctl listval |
> grep interface | awk...) and generate the configuration every N units
> of time (like every hour or every day) ?
>
>
> Aggregation of cpu-total/cpu-idle for host1-host4 and for host5-host8
> ---------------------------------------------------------------------
>
> I would make 2 blocks :
>
> <Plugin "aggregation">
> <Aggregation>
> Host host1
> Host host2
> Host host3
> Host host4
> Plugin "cpu"
> PluginInstance "cpu-total"
> Type "cpu"
> TypeInstance "idle"
>
> GroupBy "Host"
>
> CalculateSum true
> CalculateAverage true
> </Aggregation>
> <Aggregation>
> Host host5
> Host host6
> Host host7
> Host host8
> Plugin "cpu"
> PluginInstance "cpu-total"
> Type "cpu"
> TypeInstance "idle"
>
> GroupBy "Host"
>
> CalculateSum true
> CalculateAverage true
> </Aggregation>
> </Plugin>
>
> But what will be the aggregator types ? Both would result in this :
>
> global/aggregation-cpu-cpu-total-sum/cpu-idle
> global/aggregation-cpu-cpu-total-average/cpu-idle
>
>
> Update the aggregator configuration
> -----------------------------------
> Now, host1 is replaced with host11. I will of course update my
> configuration file. How will collectd take this change into account ?
> Do I have to stop/start collectd ?
>
> See the discussion I raised in with basic_aggregator, first in the
> comments of Pull Request #136 and then in the mailing list.
>
> Will plugin_register_reconfig() (or similar) be implemented soon ?
>
>
>
>
> About basic_aggregator : all these issues are solved. But I don't
> think it's a goot idea to maintain 2 similar plugins. I don't mind if
> your "wins" and mine remains as a proof of concept.
>
> Regards,
> Yves
>
>
> Le 2012-11-11 09:23, Florian Forster a écrit :
>> Hi everybody,
>>
>> the new *Aggregation plugin* has just been merged to the _master_
>> branch. With this new plugin it is possible to aggregate multiple
>> matching values into one using aggregation functions such as sum and
>> average. This allows to you create an aggregated view of the sum of
>> "cpu" metrics on each host, for example.
>>
>> Feedback and bug reports are, of course, very welcome!
>>
>> Here are some reference links:
>>
>> * An overview of the plugin in the wiki:
>> <https://collectd.org/wiki/index.php/Plugin:Aggregation>
>> * Wiki page with example configs (quite empty now, add yours!)
>> <https://collectd.org/wiki/index.php/Plugin:Aggregation/Config>
>> * Manpage entry:
>> <http://octo.cx/mancollectdaggregation>
>>
>> Best regards,
>> —octo
--
- Homepage - http://ymettier.free.fr
-
- GPG key - http://ymettier.free.fr/gpg.txt
-
- C en action - http://ymettier.free.fr/livres/C_en_action_ed2.html
-
- Guide Survie C - http://www.pearson.fr/livre/?GCOI=27440100673730
-
More information about the collectd
mailing list