<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">Progress report: I have managed to get the underlying data-structures mostly changed, I am in the process of modifying the write plugins to deal with having since source ,etric data with labels instead of the multi-source value_list_t.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">> git diffstat master<br> Makefile.am | 33 ++-<br> src/aggregation.c | 6 +-<br> src/amqp.c | 4 +-<br> src/barometer.c | 156 +++++++-----<br> src/check_uptime.c | 27 +--<br> src/csv.c | 2 +-<br> src/daemon/filter_chain.c | 30 +--<br> src/daemon/filter_chain.h | 12 +-<br> src/daemon/metrics_list_test.c | 500 +++++++++++++++++++++++++++++++++++++++<br> src/daemon/plugin.c | 802 +++++++++++++++++++++++++++++++++++++++++++++++++++-----------<br> src/daemon/plugin.h | 260 +++++++++++++++++++-<br> src/daemon/utils_cache.c | 674 +++++++++++++++++++++++++++++-----------------------<br> src/daemon/utils_cache.h | 79 ++++---<br> src/daemon/utils_cache_mock.c | 8 +-<br> src/match_value.c | 2 +-<br> src/mqtt.c | 2 +-<br> src/postgresql.c | 3 +-<br> src/target_notification.c | 2 +-<br> src/threshold.c | 12 +-<br> src/utils/cmds/getval.c | 21 +-<br> src/utils/cmds/putval.c | 2 +-<br> src/utils/common/common.c | 128 +++++++---<br> src/utils/common/common.h | 9 +-<br> src/utils/format_graphite/format_graphite.c | 2 +-<br> src/utils/format_json/format_json.c | 2 +-<br> src/utils/format_stackdriver/format_stackdriver.c | 8 +-<br> src/write_riemann_threshold.c | 4 +-<br> 27 files changed, 2123 insertions(+), 667 deletions(-)<br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"> Manoj</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 4, 2019 at 12:28 PM Manoj Srivastava <<a href="mailto:manoj.srivastava.1962@gmail.com">manoj.srivastava.1962@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace"></div><div class="gmail_default" style="font-family:monospace,monospace">Added another commit along the way:<br><br>[metric-label-support]: Modify write queue to use metric_t<br><br>This commit adds an updated write queue data structure, and updates<br>the enqueue and dequeue calling paths. Note that while this compiles<br>and the tests pass, thisis not a functioning collectd by itself, the<br>caching and networking code is a work in progress.<br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><a href="https://github.com/srivasta/collectd/tree/metric-label-support" target="_blank">https://github.com/srivasta/collectd/tree/metric-label-support</a><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Manoj</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 2, 2019 at 3:58 PM Manoj Srivastava <<a href="mailto:manoj.srivastava.1962@gmail.com" target="_blank">manoj.srivastava.1962@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">Added a commit with utility functions for cloning and freeing the new metric, identity and metadata list head data structures that will be used by the helper functions that enqueue and dequeue from the the future new write queue. I have that mostly code complete, and am writing unit tests for the new queue (well, mostly, since I have not yet written the caching functionality, and the network code will need updates too).<br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"> Manoj</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 27, 2019 at 2:21 AM Matthias Runge <<a href="mailto:mrunge@matthias-runge.de" target="_blank">mrunge@matthias-runge.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Manoj,<br>
<br>
thank you for the heads-up here. This is a great achievement.<br>
<br>
A tiny typo, your repo is this one<br>
<a href="https://github.com/srivasta/collectd/tree/metric-label-support" rel="noreferrer" target="_blank">https://github.com/srivasta/collectd/tree/metric-label-support</a><br>
<br>
Matthias<br>
<br>
On 25/11/2019 23:07, Manoj Srivastava wrote:<br>
> Hi,<br>
> <br>
> I have implemented a mechanism to convert from the current<br>
> values_list_t metric representation to the new metric_list_t<br>
> representation. This method can be called from write_queue_enque method<br>
> to convert metric data from unmigrated read plugins to provide for a<br>
> backward compatible transition path for read plugins. The next step will<br>
> be to update the storage mechanism for the write queue, and then to<br>
> update the caching and networking code.<br>
> <br>
> Comments appreciated.<br>
> <br>
> Manoj<br>
> <br>
> <br>
> Commit: <a href="https://github.com/srivasta/collectd/tree/metric-label-s" rel="noreferrer" target="_blank">https://github.com/srivasta/collectd/tree/metric-label-s</a><br>
> Design<br>
> doc: <a href="https://docs.google.com/document/d/173gGP3tUD3yfN2NNHxCv0BsKsacfDlSyoaQIn7MqLtQ/edit?usp=sharing" rel="noreferrer" target="_blank">https://docs.google.com/document/d/173gGP3tUD3yfN2NNHxCv0BsKsacfDlSyoaQIn7MqLtQ/edit?usp=sharing</a><br>
> <br>
> <br>
> On Wed, Oct 9, 2019 at 10:08 AM Manoj Srivastava<br>
> <<a href="mailto:manoj.srivastava.1962@gmail.com" target="_blank">manoj.srivastava.1962@gmail.com</a><br>
> <mailto:<a href="mailto:manoj.srivastava.1962@gmail.com" target="_blank">manoj.srivastava.1962@gmail.com</a>>> wrote:<br>
> <br>
> Hi,<br>
> <br>
> I have an initial design proposal on how to convert from a<br>
> value_list_t format to the new, single data source metrics_t format<br>
> for metrics<br>
> at <a href="https://docs.google.com/document/d/1B4drbdkPoXvMFrBx57cnKtDgeehQ1zkptq9Ozx7MnG8/edit?usp=sharing" rel="noreferrer" target="_blank">https://docs.google.com/document/d/1B4drbdkPoXvMFrBx57cnKtDgeehQ1zkptq9Ozx7MnG8/edit?usp=sharing</a><br>
> <br>
> Comments appreciated.<br>
> <br>
> Manoj<br>
> <br>
> On Mon, Sep 30, 2019 at 12:03 PM Manoj Srivastava<br>
> <<a href="mailto:manoj.srivastava.1962@gmail.com" target="_blank">manoj.srivastava.1962@gmail.com</a><br>
> <mailto:<a href="mailto:manoj.srivastava.1962@gmail.com" target="_blank">manoj.srivastava.1962@gmail.com</a>>> wrote:<br>
> <br>
> Hi,<br>
> <br>
> I have an initial stalking horse commit based on the 5.9<br>
> branch that implements the data structures in the design docs,<br>
> and has unit tests to the avl tree used in the structures. I<br>
> think we need some discussion on how to convert the current<br>
> value_list_t 5-tupples --> name + labels structure in the new<br>
> format. Also, how should we deal with metadata lists in the new<br>
> format?<br>
> <br>
> I'll try and put together the design thoughts in a document<br>
> and share that around, and perhaps a code snippet or two t go<br>
> along with it.<br>
> <br>
> Manoj<br>
> <br>
> New basic data structures and start of branch:<br>
> <a href="https://github.com/srivasta/collectd/tree/metric-label-support" rel="noreferrer" target="_blank">https://github.com/srivasta/collectd/tree/metric-label-support</a><br>
> <br>
> Adding Labels to Collectd Metrics<br>
> <a href="https://docs.google.com/document/d/173gGP3tUD3yfN2NNHxCv0BsKsacfDlSyoaQIn7MqLtQ/edit?usp=sharing" rel="noreferrer" target="_blank">https://docs.google.com/document/d/173gGP3tUD3yfN2NNHxCv0BsKsacfDlSyoaQIn7MqLtQ/edit?usp=sharing</a><br>
> Compare data structures for metric labels (collectd)<br>
> <a href="https://docs.google.com/document/d/1RZ7YAO6lWcDDvevzxFGVDgv4UQpdgWoT7OCP9Vpb2CU/edit?usp=sharing" rel="noreferrer" target="_blank">https://docs.google.com/document/d/1RZ7YAO6lWcDDvevzxFGVDgv4UQpdgWoT7OCP9Vpb2CU/edit?usp=sharing</a><br>
> -- <br>
> The Basis of the Universe Isn’t Matter or Energy—It’s Data<br>
> Manoj Srivastava <<a href="mailto:srivasta@golden-gryphon.com" target="_blank">srivasta@golden-gryphon.com</a><br>
> <mailto:<a href="mailto:srivasta@golden-gryphon.com" target="_blank">srivasta@golden-gryphon.com</a>>><br>
> <br>
> <br>
> <br>
> -- <br>
> The Basis of the Universe Isn’t Matter or Energy—It’s Data<br>
> Manoj Srivastava <<a href="mailto:srivasta@golden-gryphon.com" target="_blank">srivasta@golden-gryphon.com</a><br>
> <mailto:<a href="mailto:srivasta@golden-gryphon.com" target="_blank">srivasta@golden-gryphon.com</a>>><br>
> <br>
> <br>
> <br>
> -- <br>
> The Basis of the Universe Isn’t Matter or Energy—It’s Data<br>
> Manoj Srivastava <<a href="mailto:srivasta@golden-gryphon.com" target="_blank">srivasta@golden-gryphon.com</a><br>
> <mailto:<a href="mailto:srivasta@golden-gryphon.com" target="_blank">srivasta@golden-gryphon.com</a>>><br>
> <br>
> _______________________________________________<br>
> collectd mailing list<br>
> <a href="mailto:collectd@verplant.org" target="_blank">collectd@verplant.org</a><br>
> <a href="https://mailman.verplant.org/listinfo/collectd" rel="noreferrer" target="_blank">https://mailman.verplant.org/listinfo/collectd</a><br>
> <br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><font face="monospace, monospace">The Basis of the Universe Isn’t Matter or Energy—It’s Data</font></div><div><font face="monospace, monospace">Manoj Srivastava <<a href="mailto:srivasta@golden-gryphon.com" target="_blank">srivasta@golden-gryphon.com</a>></font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><font face="monospace, monospace">The Basis of the Universe Isn’t Matter or Energy—It’s Data</font></div><div><font face="monospace, monospace">Manoj Srivastava <<a href="mailto:srivasta@golden-gryphon.com" target="_blank">srivasta@golden-gryphon.com</a>></font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><font face="monospace, monospace">The Basis of the Universe Isn’t Matter or Energy—It’s Data</font></div><div><font face="monospace, monospace">Manoj Srivastava <<a href="mailto:srivasta@golden-gryphon.com" target="_blank">srivasta@golden-gryphon.com</a>></font></div></div></div></div></div>