[collectd] configuration: is there a way to rename/reorganize metric?

[Dudlo]

I have not received any response and I cannot see how I could force
curl_json plugin to organize the metric/values differently.

Is there a way in collectd to do a "global replace" where I change
metric/value names?

Given today's structure of data sent to Graphite:

Metrics
\---collectd
    \---serverA
        \---curl_json-serviceMetrics
                gauge-serverA-service1-requests
                gauge-serverA-service1-successes
                   ... other 8 for service1
                gauge-serverA-service2-requests
                   ... other 9 for service2
            ... and on for service3 - service10

I'd like to end up with:

Metrics
\---collectd
    \---serverA
        \---curl_json-serviceMetrics
            \---service1
                    gauge-serverA-service1-requests
                    gauge-serverA-service1-successes
                       ... other 8 for service1
            \---service2
                    gauge-serverA-service2-requests
                   ... other 9 for service2
            ... and on for service3 - service10

In other words, I'd like to "inject" the service1 to service10 into the
structure/hierarchy.

I feel that collectd's filter mechanism could be used for this but I
cannot see a clear way of doing it.

Any suggestions?

Thank you,
dudlo

---------------------------- Original Message ----------------------------
Subject: curl_json configuration question (hierarchy)
From:    "Dudlo" <collectd at mozog.com>
Date:    Fri, December 4, 2015 12:34
To:      collectd at verplant.org
--------------------------------------------------------------------------

I am using the most recent curl_json plugin for collectd. It works but I'd
like to configure it to generate more 'hierarchical' data.

I have a server that returns some performance date in this (pseudo-code)
JSON:
{
  "serverA": {
    "service1": {
      "requests":"17",
      "successes":"15",
      "failures":"2",
       ... and 7 other values ...
    }
    "service2": {
      "requests":"28",
       ... and 9 other values ...
    }
    ...
    "service20" {
      ...
    }
  }
}

My collectd.conf looks like:

LoadPlugin curl_json

<Plugin curl_json>
  <URL "http://myserver/serviceMetrics">
    Instance "serviceMetrics"
    <Key "*/*/requests">
      Type "gauge"
    </Key>
    <Key "*/*/successes">
      Type "gauge"
    </Key>
    ... and some 8 other specs
  </URL>
</Plugin>

When pushed to Graphite, this generates a "flat" list of 200 values (20
services, 10 values per service). The graphite/whisper data structure
looks like:

Metrics
\---collectd
    \---serverA
        \---curl_json-serviceMetrics
                gauge-serverA-service1-requests
                gauge-serverA-service1-successes
                   ... other 8 for service1
                gauge-serverA-service2-requests
                   ... other 9 for service2
            ... and on for service3 - service10

This is nice for a first pass and would work but it is hard to manage to
choose from 200+ items on the flat list when graphing.

I would really like a deeper hierarchy sliced by the service:

Metrics
\---collectd
    \---serverA
        \---curl_json-serviceMetrics
            +---service1
            |       gauge-serverA-service1-requests
            |       gauge-serverA-service1-successes
            |       ... other 8 for service1
            +---service2
            |       gauge-serverA-service2-requests
            |       ... other 9 for service2
            ... and on for service3 - service10

I do not know how to achieve it. I see:

*** Option 1 - doable but unpractical. Change the collectd.conf to:

<Plugin curl_json>
  <URL "http://myserver/serviceMetrics">
    Instance "serviceMetrics-service1"
    <Key "*/service1/requests">
      Type "gauge"
    </Key>
    <Key "*/service1/successes">
      Type "gauge"
    </Key>
    ... and some 8 other specs
  </URL>
  <URL "http://myserver/serviceMetrics">
    Instance "serviceMetrics-service2"
    <Key "*/service2/requests">
      Type "gauge"
    </Key>
    <Key "*/service2/successes">
      Type "gauge"
    </Key>
    ... and some 8 other specs
  </URL>
... and 8 more sections for service3 - service10
</Plugin>

There are two issues:
1. The services are dynamic. There may be 5 or 25 of them and there are no
name patterns in there (unlike the pseud-code in my example here). So I
would have to update collectd.conf with every service
addition/deletion/name change.
2. It is expensive - I'd have to redundantly call the serviceMetrics web
service 20 times (for each service), every time reading only one service
section.

*** Option 2 - impossible with curl_json plugin.

Ideally, I'd like to "force" the hierarchy on the value using the Instance
clause inside the <Key>, something like:

<Plugin curl_json>
  <URL "http://myserver/serviceMetrics">
    Instance "serviceMetrics"
    <Key "*/*/requests">
      Type "gauge"
      Instance "*/requests"
    </Key>
    <Key "*/*/successes">
      Type "gauge"
      Instance "*/successes"
    </Key>
    ... and some 8 other specs
  </URL>
</Plugin>

The 'Instance "*/requests" '  clause does not work. It simply sets the
name to the literal string (replacing special characters with
underscores). In a way, I would like to use the concept of "capturing
group" as in regex replacement. Sadly, curl_json is not ready for it.

I even tried changing the web service definition to use multi-value type,
as in load type (in types.db, load type is defined as
"shortterm:GAUGE:0:5000, midterm:GAUGE:0:5000, longterm:GAUGE:0:5000").
Again, curl_json is not ready for such data structure - it expects a
single numeric value.

Is there a way I could coerce curl_json into generating values in a
hierarchy?

Thank you,
dudlo