[collectd] [Feature Request/Version 6.0]: Adding Labels to Collectd Metrics #3094

Manoj Srivastava manoj.srivastava.1962 at gmail.com
Tue Jan 7 22:54:44 CET 2020


Progress report: I have managed to get the underlying data-structures
mostly changed, I am in the process of modifying the write plugins to deal
with having since source ,etric data with labels instead of the
multi-source value_list_t.

> git diffstat master
 Makefile.am                                       |  33 ++-
 src/aggregation.c                                 |   6 +-
 src/amqp.c                                        |   4 +-
 src/barometer.c                                   | 156 +++++++-----
 src/check_uptime.c                                |  27 +--
 src/csv.c                                         |   2 +-
 src/daemon/filter_chain.c                         |  30 +--
 src/daemon/filter_chain.h                         |  12 +-
 src/daemon/metrics_list_test.c                    | 500
+++++++++++++++++++++++++++++++++++++++
 src/daemon/plugin.c                               | 802
+++++++++++++++++++++++++++++++++++++++++++++++++++-----------
 src/daemon/plugin.h                               | 260
+++++++++++++++++++-
 src/daemon/utils_cache.c                          | 674
+++++++++++++++++++++++++++++-----------------------
 src/daemon/utils_cache.h                          |  79 ++++---
 src/daemon/utils_cache_mock.c                     |   8 +-
 src/match_value.c                                 |   2 +-
 src/mqtt.c                                        |   2 +-
 src/postgresql.c                                  |   3 +-
 src/target_notification.c                         |   2 +-
 src/threshold.c                                   |  12 +-
 src/utils/cmds/getval.c                           |  21 +-
 src/utils/cmds/putval.c                           |   2 +-
 src/utils/common/common.c                         | 128 +++++++---
 src/utils/common/common.h                         |   9 +-
 src/utils/format_graphite/format_graphite.c       |   2 +-
 src/utils/format_json/format_json.c               |   2 +-
 src/utils/format_stackdriver/format_stackdriver.c |   8 +-
 src/write_riemann_threshold.c                     |   4 +-
 27 files changed, 2123 insertions(+), 667 deletions(-)

  Manoj

On Wed, Dec 4, 2019 at 12:28 PM Manoj Srivastava <
manoj.srivastava.1962 at gmail.com> wrote:

> Added another commit along the way:
>
> [metric-label-support]: Modify write queue to use metric_t
>
> This commit adds an updated write queue data structure, and updates
> the enqueue and dequeue calling paths. Note that while this compiles
> and the tests pass, thisis not a functioning collectd by itself, the
> caching and networking code is a work in progress.
>
> https://github.com/srivasta/collectd/tree/metric-label-support
>
> Manoj
>
> On Mon, Dec 2, 2019 at 3:58 PM Manoj Srivastava <
> manoj.srivastava.1962 at gmail.com> wrote:
>
>> Added a commit with utility functions for cloning and freeing the new
>> metric, identity and metadata list head data structures that will be used
>> by the helper functions that enqueue and dequeue from the the future new
>> write queue. I have that mostly code complete, and am writing unit tests
>> for the new queue (well, mostly, since I have not yet written the caching
>> functionality, and the network code will need updates too).
>>
>>    Manoj
>>
>> On Wed, Nov 27, 2019 at 2:21 AM Matthias Runge <mrunge at matthias-runge.de>
>> wrote:
>>
>>> Hi Manoj,
>>>
>>> thank you for the heads-up here. This is a great achievement.
>>>
>>> A tiny typo, your repo is this one
>>> https://github.com/srivasta/collectd/tree/metric-label-support
>>>
>>> Matthias
>>>
>>> On 25/11/2019 23:07, Manoj Srivastava wrote:
>>> > Hi,
>>> >
>>> >   I have implemented a mechanism to convert from the current
>>> > values_list_t metric representation to the new metric_list_t
>>> > representation. This method can be called from write_queue_enque method
>>> > to convert metric data from unmigrated read plugins to provide for a
>>> > backward compatible transition path for read plugins. The next step
>>> will
>>> > be to update the storage mechanism for the write queue, and then to
>>> > update the caching and networking code.
>>> >
>>> >     Comments appreciated.
>>> >
>>> >     Manoj
>>> >
>>> >
>>> >  Commit: https://github.com/srivasta/collectd/tree/metric-label-s
>>> > Design
>>> > doc:
>>> https://docs.google.com/document/d/173gGP3tUD3yfN2NNHxCv0BsKsacfDlSyoaQIn7MqLtQ/edit?usp=sharing
>>> >
>>> >
>>> > On Wed, Oct 9, 2019 at 10:08 AM Manoj Srivastava
>>> > <manoj.srivastava.1962 at gmail.com
>>> > <mailto:manoj.srivastava.1962 at gmail.com>> wrote:
>>> >
>>> >     Hi,
>>> >
>>> >       I have an initial design proposal on how to convert from a
>>> >     value_list_t format to the new, single data source metrics_t format
>>> >     for metrics
>>> >     at
>>> https://docs.google.com/document/d/1B4drbdkPoXvMFrBx57cnKtDgeehQ1zkptq9Ozx7MnG8/edit?usp=sharing
>>> >
>>> >       Comments appreciated.
>>> >
>>> >       Manoj
>>> >
>>> >     On Mon, Sep 30, 2019 at 12:03 PM Manoj Srivastava
>>> >     <manoj.srivastava.1962 at gmail.com
>>> >     <mailto:manoj.srivastava.1962 at gmail.com>> wrote:
>>> >
>>> >         Hi,
>>> >
>>> >             I have an initial stalking horse commit based on the 5.9
>>> >         branch that implements the data structures in the design docs,
>>> >         and has unit tests to the avl tree used in the structures. I
>>> >         think we need some discussion on how to convert the current
>>> >         value_list_t 5-tupples --> name + labels structure in the new
>>> >         format. Also, how should we deal with metadata lists in the new
>>> >         format?
>>> >
>>> >           I'll try and put together the design thoughts in a document
>>> >         and share that around, and perhaps a code snippet or two t go
>>> >         along with it.
>>> >
>>> >            Manoj
>>> >
>>> >         New basic data structures and start of branch:
>>> >         https://github.com/srivasta/collectd/tree/metric-label-support
>>> >
>>> >         Adding Labels to Collectd Metrics
>>> >
>>> https://docs.google.com/document/d/173gGP3tUD3yfN2NNHxCv0BsKsacfDlSyoaQIn7MqLtQ/edit?usp=sharing
>>> >         Compare data structures for metric labels (collectd)
>>> >
>>> https://docs.google.com/document/d/1RZ7YAO6lWcDDvevzxFGVDgv4UQpdgWoT7OCP9Vpb2CU/edit?usp=sharing
>>> >         --
>>> >         The Basis of the Universe Isn’t Matter or Energy—It’s Data
>>> >         Manoj Srivastava <srivasta at golden-gryphon.com
>>> >         <mailto:srivasta at golden-gryphon.com>>
>>> >
>>> >
>>> >
>>> >     --
>>> >     The Basis of the Universe Isn’t Matter or Energy—It’s Data
>>> >     Manoj Srivastava <srivasta at golden-gryphon.com
>>> >     <mailto:srivasta at golden-gryphon.com>>
>>> >
>>> >
>>> >
>>> > --
>>> > The Basis of the Universe Isn’t Matter or Energy—It’s Data
>>> > Manoj Srivastava <srivasta at golden-gryphon.com
>>> > <mailto:srivasta at golden-gryphon.com>>
>>> >
>>> > _______________________________________________
>>> > collectd mailing list
>>> > collectd at verplant.org
>>> > https://mailman.verplant.org/listinfo/collectd
>>> >
>>>
>>>
>>
>> --
>> The Basis of the Universe Isn’t Matter or Energy—It’s Data
>> Manoj Srivastava <srivasta at golden-gryphon.com>
>>
>
>
> --
> The Basis of the Universe Isn’t Matter or Energy—It’s Data
> Manoj Srivastava <srivasta at golden-gryphon.com>
>


-- 
The Basis of the Universe Isn’t Matter or Energy—It’s Data
Manoj Srivastava <srivasta at golden-gryphon.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20200107/b2f5034e/attachment-0001.html>


More information about the collectd mailing list