[collectd] collectd rrdtool performance

[Thorsten von Eicken]

I posted over a month ago about my performance woes with
collectd/rrdtool. It boils down to the fact that I'm now collecting
stats for 290 hosts resulting in 20500 RRDs being written every 10
seconds. I'm doing this is on a dual-core Amazon EC2 instance with 8GB
memory and two 420GB spindles. When striped, the disks do at most about
600-800 random writes per second regardless of which filesystem I use
(xfs, riser, ext2, ext3). From past measurements, I'm attempting
probably 3000 random writes/sec or more, which way exceeds the disk speeds.

When I had half the hosts to collect the system would work, but the disk
writes would lag several minutes behind, i.e., after killing collectd
the disks would continue rattling for 2-3 minutes at 100% util. Of
course in a state like that the system is virtually unusable for
anything else, like generating graphs. I realized I needed some scheme
that would convert the random writes to sequential writes. In addition I
needed a backup strategy regardless in case the EC2 server dies.

After some thinking and experimentation I did find a solution that seems
to work incredibly well: tmpfs! I created a 500GB partition striped
across the two drives, mounted it on swap, and created a (just short of)
500GB tmpfs. That's where I store the RRDs. I also run a backup process
that frequently pushes modified RRDs to Amazon S3 to minimize data loss
in case of machine failure. (An alternative backup is forwarding the
data through collectd to a second server, which I did for a while and
which works very well too.)

The tmpfs set-up has exceeded all my expectations. The machine is 80%
idle and spends only 2% in IO wait. Everything is responsive as long as
I don't run some memory hungry app. The total RRDs stored occupy 261GB
and the active set of RRDs occupies 8.7GB. The app rendering the graphs
works fine too, even though it obviously has to make random reads. I've
been running this set-up for several weeks now am I'm still wondering
when it will break down. (Yes, I'll report :-)

Anyway, I don't recommend everyone switch to tmpfs, but if you need to
do 5x more random writes than your disks support, consider tmpfs!

Thorsten - www.rightscale.com - Amazon EC2 deployments made easy!

Florian Forster wrote:
> Hi Thorsten,
>> The biggest issue I see is that 150 hosts = 10500 RRDs. I'm planning
>> to go ahead and reorganize a little how the RRD data is stored by
>> placing all related variables of a plugin into a single RRD as opposed
>> to the current scheme where almost every variable is in its own RRD.
> 
> Hm, I'd do some benchmarks first. The effect is that the DSes are stored
> nearby, basically the same effect the `CacheTimeout' option uses.