[collectd] Averaging Samples when SigRok provides higher resolution

[Kieran Bingham]

Hi Octo!

On 25 September 2013 21:39, Florian Forster <octo at collectd.org> wrote:
> Hi Kieran,
>
> On Wed, Sep 25, 2013 at 03:22:33PM +0100, Kieran Bingham wrote:
>> The Multimeter has an update rate of 5Hz, but we are storing the data
>> as RRD, which appears to be limited as 1Hz by its data storage format.
>
> that's true, altough I don't know the exact reason for this from the top
> of my head. RRDtool will allow multiple readings for a "primary data
> point" (PDP), but they have to have increasing timestamps attached. So
> if you have "step" (interval) set to 10 seconds, you can, in theory, add
> ten values to form a PDP. For this reason, specifying a consolidation
> function for RRAs with "step=1" actually does make some sense.
>

I think it comes from the database format used by RRDTool storing
offsets in seconds?

> It might be that the strict "ts_old < ts_new" ordering is not really
> _required_ for gauges and could be relaxed to "ts_old <= ts_new", but
> again, just rambling from the top of my head.
>
>> To enhance the data for our graphs, it was suggested to store the
>> average of all of the readings per dispatch
>
> I think it might be interesting to add something like this, though I
> would prefer a more generic approach. Again stolen from RRDtool:
>
>   Consolidation min|max|average|last
>
> Where "last" would be the current behavior, "average" the one you're
> aiming at. This way you can adapt this to your needs, e.g. supply chain
> cooling might want to use a "max" aggregation over the temperature
> readings.

Agreed - and in fact the min/max will come in useful for us too.
I had aimed to make this configurable when I started, so with your
prompting - I've changed the 'average_guage' to a consolidation_value.


This fits our needs quite nicely for the moment, and patch is available
at github on the link below if its useful to anyone else:

Berts concerns about averaging the data may be valid - so I guess its up
to you guys to decide if you want to integrate this in any form to the mainline.

The new patch does provide more options than just averaging, so
if other peoples use-cases differ - then there may be a use for them too.

As it's configurable and selectable it shouldn't be harmful at least...



https://github.com/kbingham/collectd/commit/c3160d0622fd174d4bcbae954e3877e517df634f

sigrok plugin: Provide consolidation configuration

If the requested update rate is less than the refresh rate of the
device, we are losing precision and data. Instead of taking only
a sample every period - we can consolidate this data into a single
sample point.

There are 4 methods of consolidation:

  Consolidation min|max|average|last

  min:     The smallest sample in the period is used
  max:     The largest sample point is taken
  average: An average of all samples over the period is provided.
  last:    The default method, where the last sample point is given

This will ensure that the data precision taken by the samples are
not lost, but can be used and reported in a user configurable way

To use this feature, provide the Consolidation option in the
<Device> configuration section as shown by the following example:

<Plugin sigrok>
  LogLevel 3
  <Device "inlet_current">
    Driver "agilent-dmm"
    MinimumInterval 10
    Consolidation average
    Conn "/dev/ttyUSB0"
  </Device>
</Plugin>
----------------------------------------------------------------
Kieran Bingham (1):
      sigrok plugin: Provide consolidation configuration

 src/sigrok.c | 118
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 115 insertions(+), 3 deletions(-)

Thanks and Regards

Kieran