[collectd] RRDs on SSD - performance degredation [resend without images]

[Jesse Reynolds]

Hello, 

We're noticing some interesting patterns in collectd / librrd is flushing RRD files to disk. 

We have a 1TB volume for RRD files, which is at 75% capacity. Over the last few weeks the response time of the disks has slowly degraded, with a few sudden jumps (see graph below). When it jumps up in response time, it seems collectd responds by halving the write ops, and octets. Is it just deciding to flush half as often? If anyone has some pointers to the details of this behaviour, that'd be great. 

Also, does anyone recognise this pattern as being related to how full the SSD backed volume is, and perhaps the disks are now needing to do garbage collection? I'm pretty new to SSDs, and I suspect RRDs are a fairly uncommon way to utilise SSD drives. 

FWIW, we have 5 x 256 GB Micron SSDs in a IBM hardware RAID 5 set. The OS is linux (ubuntu 12.04). 

disk time, ms: 
   https://dl.dropboxusercontent.com/u/5932161/visage/disk%20time%20ms%201%20month.png

bytes per second: 
   https://dl.dropboxusercontent.com/u/5932161/visage/rrd%20disk%20octets%20per%20second%201%20month.png

RRD filesystem disk utilisation: 
   https://dl.dropboxusercontent.com/u/5932161/visage/rrd%20disk%20utilisation%201%20mth.png

Thank you!
Jesse