[collectd] Collectd Best Know Deployment Methods

[Matthias Runge]

On 02/04/2020 02:32, Abraham A wrote:
>> I'm studiyng a similar scenario (ceilometer+gnocchi) but only for hypervisor
>> metrics (we currently use collectd in our hypervisors and cloud servers, and
>> send metrics to graphite).
>>
>> I'm guessing you could install collectd in your cloud servers and send the
>> metrics to a (maybe sepparate) collectd instance acting as a forwarder to
>> gnocchi.
>>
>> That way you can use a collectd password specific to the cloud servers without
>> exposing your keystone credentials.
> 
> Thanks Ricardo for your suggestion,
> 
> I will set up that intermediate instance acting as a forwarder using
> the Network type plugin
> for now, what I understand so far is the following high level deployment:
> 
> 1. Bare Metal / Virtual Instance: Plugin:Network > Client (Write Type Plugin)
> 2. Intermediate Instance / Forwarder with 2 plugins:
>     - Plugin:Network > Server (Read Type Plugin)
>     - Plugin:Collectd Gnocchi Python (Write Type Plugin)
> 
> I assume in that intermediate instance, collectd takes care of
> grabbing the input from the
> network plugin and sending it through any of the configured write type
> plugins, in this case,
> the collect gnocchi plugin.
> 
> I will report back what was the result of this.
> 

Hi,

there are a few caveats here.
- you are using only one controller, there is no HA.
- I assume, you have gnocchi installed on your controller as well,
again, no HA.
- make sure, you have a performant gnocchi backend, i.e. separate ceph
nodes.
- when using swift as gnocchi backend, that can lead to starvation of
the controller.
- for simplest collectd setups, you can surely use the network plugin to
forward metrics between collectds.


You've mentioned, you are using Stein. We implemented in Train (and
backported to Queens) the qdr mesh using Apache QPID. Collectd connects
to the AMQP1 bus via the amqp1 plugin. On the other side, there is a
component named "Smart Gateway", which provides a scrape endpoint for
Prometheus and Elasticsearch to store metrics and also events.

More details can be found on
https://infrawatch.github.io/documentation/


Matthias