<div dir="ltr">Hi list,<div><br></div><div>First of all, shout-out to octo and jeremy katz for making the hackathon happen, great stuff and a great opportunity to meet you all.</div><div><br></div><div>Here are my answers and comments:</div>
<div><br></div><div>1) OK. Fully in favor, I don't think the extra disk space will be much of a problem, it will greatly simplify the API.</div><div>2) OK. My gut initially said no, but rather because I hadn't wrapped my head around the fact that gauge still was there and provided all necessary information. I churned to find use cases where this would be interesting to have.</div>
<div>3) I'm am strongly in favor of solution 2, because it is the one that would allow the most flexible way of interacting with other outputs than rrdtool and graphite. Resolving to something ressembling a path name is a task that concerns mostly:</div>
<div> - the csv output plugin</div><div> - the rrd output plugin</div><div> - the write_graphite output plugin</div><div><br></div><div>I think there is a way to make this work out for these plugins as well as discussed saturday.</div>
<div><br></div><div>The proposed way of doing it was to have plugins hint at the way a name could be construed. The clear advantage of this is approach is that an internal mangling DSL could use the fields and</div><div>it would ease interop with tools such as riemann, logstash or librato.</div>
<div><br></div><div>Serialisation is another debate :)</div><div><br></div><div>Cheers,</div><div> - pyr</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Sep 23, 2013 at 8:12 PM, Florian Forster <span dir="ltr"><<a href="mailto:octo@collectd.org" target="_blank">octo@collectd.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">[TLDR: Do you have a use-case for raw counter values?]<br>
<br>
Good morning everybody,<br>
<br>
we had a great time at the Hackathon [0] in Berlin yesterday. Thanks<br>
again to everyone!<br>
<br>
Amongst the ideas we discussed were some fundamental changes to the way<br>
metrics are represented. These ideas might eventually result in a<br>
collectd version 6, but hold you breath just yet – no actual coding has<br>
been done in that direction, we're just collecting design ideas at the<br>
moment.<br>
<br>
<br>
1) Get rid of multiple "data sources" per metric.<br>
<br>
Some metrics, e.g. the "if_octets" metrics from the "interface" plugin<br>
and the "load" metric from the "load" plugin have multiple "data<br>
sources". The "if_octets" metrics has data sources "rx" and "tx" for<br>
received and transmitted bytes.<br>
<br>
We would like to remove this functionality altogether. Rather than one<br>
metric with two values, we would like the "interface" plugin to create<br>
two metrics with one value each. Since version 5.0 this is mostly how<br>
metrics are defined and only few cases are left, now we would like to<br>
actually remove the functionality. We reached a consensus on this so<br>
it's essentially a done deal.<br>
<br>
Pro:<br>
<br>
* A lot of collectd code becomes a lot easier (less bugs)<br>
* A lot of front-end and graphing code becomes a lot easier (more<br>
and better front-ends)<br>
* Mapping of collectd metrics to names used by other systems,<br>
e.g. Graphite, is easier / more consistent<br>
* Splitting up existing RRD files by "data source" is a solved<br>
problem; writing a migration script is fairly simple<br>
* A point which causes much confusion for new users is resolved<br>
<br>
Contra:<br>
<br>
* Building a backwards compatibility layer for this is going to be<br>
hard<br>
<br>
<br>
2) Calculate the rate of counters / DERIVEs early on and after that only<br>
handle gauge values.<br>
<br>
Right now, values come in four flavors: GAUGE and DERIVE, and two more<br>
special cases which are hardly ever used. These numbers are passed<br>
through the daemon as they are, i.e.:<br>
* The CPU plugin gets a counter of how many ticks / jiffies the CPU<br>
has spent in user mode since some unspecified time in the past.<br>
* This number if "dispatched" as a DERIVE type value.<br>
* The output plugins will write this absolute number.<br>
<br>
However, in the case of DERIVE (and COUNTER) values these actual<br>
absolute numbers are meaningless. In order to do anything meaningful<br>
with them, the difference between two values (and their respective<br>
times) is calculated, which results in the averaged _rate_ of change.<br>
This is what output plugins do if they have an enabled "StoreRates"<br>
setting. But not only there: Threshold checking, scaling, aggregation;<br>
all of these operate on the _rate_ rather than the absolute number.<br>
<br>
We would like to change the way DERIVEs are handled within collectd:<br>
Instead of keeping the original absolute values, we would like to<br>
calculate the rate as early as possible, possibly within the read<br>
plugins, and only handle the rate form there on.<br>
<br>
We only came up with one use case where having the raw counter values is<br>
beneficial: If you want to calculate the average rate over arbitrary<br>
time spans, it's easier to look up the raw counter values for those<br>
points in time and go from there. However, you can also sum up the<br>
individual rates to reach the same result. Finally, when handling<br>
counter resets / overflows within this interval, integrating over /<br>
summing rates is trivial by comparison.<br>
<br>
Do you have any other use-case for raw counter values?<br>
<br>
Pro:<br>
<br>
* Handling of values becomes easier.<br>
* The rate is calculated only once, in contrast to potentially several<br>
times, which might be more efficient (currently each rate conversion<br>
involves a lookup call).<br>
* Together with (1), this removes the need for having the "types.db",<br>
which could be removed then. We were in wild agreement that this<br>
would be a worthwhile goal.<br>
<br>
Contra:<br>
<br>
* Original raw value is lost. It can be reconstructed except for a<br>
(more or less) constant offset, though.<br>
<br>
<br>
3) Changes to the naming schema.<br>
<br>
This we discussed the most and the most diverse. Currently, collectd has<br>
a very static naming schema consisting of host, plugin, type and two<br>
optional fields, "plugin instance" and "type instance". This works well<br>
in many cases, but has some drawbacks and limitations. For example, the<br>
Varnish plugin puts the Varnish server and the subcomponent into the<br>
"plugin instance", which is not ideal.<br>
<br>
We discussed two alternatives:<br>
<br>
* Use a path (or, expressed more sciency, an ordered list of strings)<br>
to identify metrics. A CPU metric could look like this:<br>
<br>
"/<a href="http://example.com/cpu/0/idle" target="_blank">example.com/cpu/0/idle</a>"<br>
<br>
* Use an (unordered) set of key-value pairs to identify metrics. You<br>
can think of this as a JSON object that only has string members, if<br>
you like. We would likely make at least two fields mandatory, for<br>
example "source" (or "host") and "metric" (or "name"). A CPU metric<br>
could looke like this, for example:<br>
<br>
{<br>
"source": "<a href="http://example.com" target="_blank">example.com</a>", // required<br>
"metric": "cpu usage", // required<br>
"cpu-id": "0", // optional<br>
"cpu-state": "idle" // optional<br>
}<br>
<br>
When filtering or aggregating, the first option would require to use<br>
indexes, for example "get metrics where index 0 is '<a href="http://example.com" target="_blank">example.com</a>'". Here,<br>
"index 0" refers to the "source". The second alternative would allow us<br>
to use names (rather than indexes) to refer to a specific part of the<br>
name, e.g. "get metrics where 'source' is '<a href="http://example.com" target="_blank">example.com</a>'".<br>
<br>
I'd love to hear what people think about the entire topic of naming.<br>
There are good reasons for either schema and there are also good reasons<br>
for staying with the current concept and live with its flaws. Which<br>
schema would meet your needs best and why? What are those needs?<br>
<br>
Best regards,<br>
—octo<br>
<br>
<br>
[0] <<a href="https://collectd.org/wiki/index.php/Hackathon_2013_Berlin" target="_blank">https://collectd.org/wiki/index.php/Hackathon_2013_Berlin</a>><br>
<span class="HOEnZb"><font color="#888888">--<br>
collectd – The system statistics collection daemon<br>
Website: <a href="http://collectd.org" target="_blank">http://collectd.org</a><br>
Google+: <a href="http://collectd.org/+" target="_blank">http://collectd.org/+</a><br>
GitHub: <a href="https://github.com/collectd" target="_blank">https://github.com/collectd</a><br>
Twitter: <a href="http://twitter.com/collectd" target="_blank">http://twitter.com/collectd</a><br>
</font></span><br>-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v1.4.10 (GNU/Linux)<br>
<br>
iQIcBAEBAgAGBQJSQISTAAoJEFCaZHVExRxoE3kP/10l+4xZW5Fxnu0joUESJJnP<br>
XSeC+SPy+i84jqgeB03Rmyy4GEyjqfHfrGaVwF56OiivhUgisU7DtiCppCyOl4aA<br>
xBM6z1L21QAcxgKh1VEfZJdhQtO1g3lyBK+ofA+S7RbU/k4+TNGeztx6dFpqoypJ<br>
wwtowpyLSsGbCzUmMeibWRjjuUhPrK0M8hnrlR9zG0wi3+Jp31wUfK0eKwcB4LIU<br>
aOPlu3Yp1JFmSNpfPEKS9ZAc6tmRvEqyOpRC6ujidEd/8+qGtgLaADfh9KxgTzwv<br>
6BtufwXQ+a/g31Gbs0pg0JZp1XCIwJ9EtBN3FdfM2pSThMnrrIBRiveZ/1Oh92uk<br>
z3mEiZOR/b8Np8G6Xu50TC8zZU7mO1kVlbCAN1e+pJZn0sdzmctc3tRBbASe77CF<br>
BLiVQ/XpQe037DJrwFCnfPgxqbPxf6tGsDyhIW3gf5aSQUUIOlTppqVmfds+8NhM<br>
7B87YxCuXa9XHNZSjZT6F3iUAqbGhye5Niwn4U39DPmM5dfT4ufwsfUJBpOhD2JK<br>
T/WUCZZJQ006yW0BYTHinn3eyU3tlIwWiamqw7YQq4DPIHvB/AtT4AeMRb0x2rUZ<br>
36HFXWhFTsq6FzRS/0KGPly0SLBPlz63ibY3vdpJYDMlqphF163T8jt50HMaCzzY<br>
2gcNwILoHn7InuXfnKXt<br>
=XJpV<br>
-----END PGP SIGNATURE-----<br>
<br>_______________________________________________<br>
collectd mailing list<br>
<a href="mailto:collectd@verplant.org">collectd@verplant.org</a><br>
<a href="http://mailman.verplant.org/listinfo/collectd" target="_blank">http://mailman.verplant.org/listinfo/collectd</a><br>
<br></blockquote></div><br></div>