[collectd] Differential data for if_octets

[Giovanni Torres]

On Tue, Jan 19, 2016 at 5:18 PM Pawel Akonom <pawel.akonom at vnomic.com>
wrote:

> Hi,
>
> I am trying to visualize some system metrics on grafana. Metrics are
> sent from CollectD to Logstash, then from Logstash to Elasticsearch and
> from Elasticsearch to Grafana. The problem is rx tx if_octets and
> if_packets metric is growing all the time. I would like to see
> differential metric on the graph - how many octets or packets was send
> and received in last 10 seconds.
>
> I am using collectd 5.5 on CentOS 6.7 and I have compiled it from source
> code. In /usr/share/collectd/types.db if_packets and if_octets are set
> to DERIVE. As I understand from wiki page
> https://collectd.org/wiki/index.php/Data_source this data should be sent
> as a change from the last time:
>
> [root at hdptest-mster2 collectd-5.5.0]# grep if_packets
> /usr/share/collectd/types.db
> if_packets              rx:DERIVE:0:U, tx:DERIVE:0:U
> [root at hdptest-mster2 collectd-5.5.0]# grep if_octets
> /usr/share/collectd/types.db
> if_octets               rx:DERIVE:0:U, tx:DERIVE:0:U
>
> Here is my configuration file:
>
> [root at hdptest-mster2 collectd-5.5.0]# egrep -v ^#.* /etc/collectd.conf |
> sed '/^$/d'
> Hostname    "ACI_184_vora2_Master_MT2"
> TypesDB     "/usr/share/collectd/types.db"
> LoadPlugin cpu
> LoadPlugin df
> LoadPlugin disk
> LoadPlugin exec
> LoadPlugin interface
> LoadPlugin load
> LoadPlugin memory
> LoadPlugin network
> <Plugin cpu>
>    ReportByCpu true
>    ReportByState true
>    ValuesPercentage true
> </Plugin>
> <Plugin df>
>          Device "/dev/sda1"
>          Device "/dev/sdb1"
>          FSType "ext4"
>          ValuesPercentage true
> </Plugin>
> <Plugin disk>
>          Disk "/^sd[a-b]?$/"
>          IgnoreSelected false
> </Plugin>
> <Plugin interface>
>          Interface "eth0"
>          IgnoreSelected false
> </Plugin>
> <Plugin memory>
>          ValuesAbsolute true
>          ValuesPercentage false
> </Plugin>
> <Plugin network>
>          <Server "192.168.112.228" "25826">
>          </Server>
>          <Server "192.168.112.227" "25826">
>          </Server>
> </Plugin>
>
> This is a sample of raw data sent to logstash from this host:
>
> [root at hdptest-elk logstash]# tail -f logstash-rawdata.log | grep MT2 |
> grep if_octets
>
> {"host":"ACI_184_vora2_Master_MT2","@timestamp":"2016-01-19T21:37:41.485Z","plugin":"interface","collectd_type":"if_octets","plugin_instance":"eth0","rx":4202999326,"tx":1272176172,"@version":"1","type":"collectd"}
>
> {"host":"ACI_184_vora2_Master_MT2","@timestamp":"2016-01-19T21:37:51.485Z","plugin":"interface","plugin_instance":"eth0","collectd_type":"if_octets","rx":4203048744,"tx":1272200296,"@version":"1","type":"collectd"}
>
> {"host":"ACI_184_vora2_Master_MT2","@timestamp":"2016-01-19T21:38:01.485Z","plugin":"interface","collectd_type":"if_octets","plugin_instance":"eth0","rx":4203131562,"tx":1272215389,"@version":"1","type":"collectd"}
>
> {"host":"ACI_184_vora2_Master_MT2","@timestamp":"2016-01-19T21:38:11.485Z","plugin":"interface","collectd_type":"if_octets","plugin_instance":"eth0","rx":4203152162,"tx":1272241466,"@version":"1","type":"collectd"}
>
> It looks like data are sent as a GAUGE value instead of DERIVE. What I
> am doing wrong? How can I configure CollectD to send differential
> statistics for the interfaces?
>
> Best Regards,
> Pawel
>
>
> What is your logstash configuration?  I'm assuming that logstash is
configured to listen on udp/25826 and you are using the network plugin as a
write plugin to this port?

I don't have the answer to your question, but I have a similar setup and I
see the same issue, where grafana renders the datapoints as a counter,
rather than deltas between successive datapoints.

>From what I understand about collectd, it will cache values in the global
cache (https://collectd.org/wiki/index.php/Global_cache) where DERIVE
values are converted to a rate.

The interface plugins collects values from /proc/net/dev and sends them to
the network plugin on the client as is.  It is here, I believe, that the
network plugin sends the values to the logstash listener, if that is how
you have it setup, again with the values intact and not as a rate.  This is
why grafana renders them as such.

In a typical setup, many clients will use the network plugin to send plugin
datapoints to a server instance running the network plugin in Listening
mode.  From there, an output plugin is configured to send to somewhere
else, like write_graphite, for example.  If you are familiar with graphite,
the datapoints first go to carbon.  If you run ngrep on the carbon port,
you would see that the values for if_octets and if_packets are the rate
values (I assume from the global cache) and not the raw DERIVE values that
continue to grow.

It's almost like the network plugin, when configured to Listen, should have
an option like StoreRates=True, like some of the other plugins, but I don't
think the network plugin was intended for this.

Giovanni
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20160125/c3ec4ec6/attachment.html>