[collectd] Problems with the curl_json plugin an ElasticSearch 'node stats' info?

[Matt Wise]

Hey we're still unable to solve this problem... I was wondering if anyone
has any thoughts? I've tried narrowing the stats down so that the JSON
being returned looks this simple:

{
>     "cluster_name": "nsp-es-staging-us1",
>     "nodes": {
>         "zDQZnKxxQPC1uOe7qZYHcA": {
>             "timestamp": 1449605835281,
>             "name":
> "staging-us1-es-access-uswest2-5-i-bd52dd64-nsp-es-staging-us1",
>             "transport_address": "inet[/10.32.17.234:9300]",
>             "host": "xxx,
>             "ip": [
>                 "inet[/10.32.17.234:9300]",
>                 "NONE"
>             ],
>             "attributes": {
>                 "aws_availability_zone": "us-west-2a",
>                 "data": "false",
>                 "master": "true"
>             },
>             "http": {
>                 "current_open": 8,
>                 "total_opened": 114
>             }
>         }
>     }
> }


and  with this very simple elasticsearch.conf file:

LoadPlugin curl_json
> <Plugin curl_json>
>   <URL "http://127.0.0.1:9200/_nodes/_local/stats/http?pretty=true">
>     Instance "elasticsearch"
>     <Key "nodes/zDQZnKxxQPC1uOe7qZYHcA/http/current_open">
>       Type "gauge"
>       Instance "http_current_open"
>     </Key>
>   </URL>
> </Plugin>


And we're *still* not even getting that metric to come through the system..
can anyone help here?


Matt Wise
Sr. Systems Architect
Nextdoor.com

On Wed, Aug 5, 2015 at 12:53 PM, Matt Wise <matt at nextdoor.com> wrote:

> In our environment, we're using Collectd 4.10.1 (I know ... old as dirt,
> but its a requirement of RightScale) and we are trying to leverage the
> curl_json plugin to pull metrics from our ElasticSearch cluster. We have
> the metrics coming in properly from http://localhost:9200/_cluster/stats
> and http://localhost:9200/_cluster/health.
>
> What we can't get working though are *any* metrics from
> http://localhost:9200/_nodes/_local/stats. No matter how many or few
> metrics I define, and no matter whether I use 'nodes/*/...' or
> 'nodes/<insert node id here>/...', I cannot seem to get any data from this
> endpoint.
>
> Here is our entire collectd elasticsearch.conf file, and a real copy of
> the raw node stats json output:
>   https://gist.github.com/diranged/ac7d2edb45b0eba00d42
>
> Any ideas on what might be going on? I can't seem to get any debug logs
> either, and unfortunately the collectd-dbg package for 4.10.1 is not
> available it seems like, so we're really hamstrung here.
>
> I have verified that ..
>   .. Collectd is indeed making the curl call to ElasticSearch and the
> response is coming back properly (via strace, and tcpdump)
>  .. Replacing '*' with 'HYUqTMnfSgmvd87_o3UPLw' (the node id) in the
> config file does not help
>  .. Dropping all the way down to a single metric
> (jvm/mem/heap_used_in_bytes) does not help
>
> Can anyone help here?
>
> Matt Wise
> Sr. Systems Architect
> Nextdoor.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20151208/279fa006/attachment-0001.html>