[collectd] RRDs on SSD - performance degredation [resend without images]

[Florian Forster]

Hi Jesse,

On Fri, Aug 16, 2013 at 06:49:25PM +0930, Jesse Reynolds wrote:
> Sounds too good to be true :-)

collectd's "rrdtool" plugin and RRDCacheD use the same basic principle:
Cache updates to RRD files in memory and write multiple updates at once.
Due to the way RRD files are laid out, this will result in a sequential
write.

On spinning disks, updated to RRD files are dominated by disk seeks. By
batching updates, you get _way_ better performance. As in, updating one
metric vs. 32 metrics boils down to the same number of disk seeks. In
contrast to this, the amount of data being written, at the scale of
metrics written to RRD files, does not matter. Whether you write one
metric (8 bytes) or 32 metrics (256 bytes, 1-2 blocks) or 1024 metrics
(8 kByte, 32-33 blocks) doesn't make a difference in disk latency.

On SSDs, "seeks" don't exist in the same way they do on spinning media,
but writing sequentially still has benefits. My understanding of SSDs is
based on hear-say, so please take this with a grain of salt. As far as I
know, one bit transition, e.g. from 1 to 0 is easy for Flash, the other
way around, i.e. from 0 to 1, is _not_. As in, you can only reset an
entire block to all 1's and then flip individual bits to 0. These blocks
are quite large, in the 4 kByte ballpark. Since this is slow, the entire
block is usually written to somewhere else and the previous copy blanked
in the background. Again, sequential writes are a big advantage, but for
entirely different reasons.

Although the principle is the same, there are differences between the
"rrdtool" plugin and RRDCacheD: The plugin can write at a configured
rate (e.g. 50 writes per second), the daemon can write a journal and
does not need to write all state to disk when shutting down. The biggest
advantage for RRDCacheD, though, is that it allows you to restart
collectd without flushing everything to disk.

> Is the scheduling improved somehow? Is it more efficient with writing
> the RRD files? It seems that it will still need to update the RRD
> files as the rrd plugin does now, so not sure how this means there'll
> be less demands placed on IO. 

I/O wise, the two should be the same. If not, I'd love to know why and
back-port any performance improvements to the "rrdtool" plugin.

Back to the problem at hand, I have a theory:

I noticed that there is an increase in used disk space whenever the
performance degrades. That is usually caused by new RRD files being
created. This used to block the "rrdtool" plugin, leading to less normal
updates per second being performed. Add the long wait for file creation
and you have an explanation for the average wait time increasing. You
can try the "CreateFilesAsync" option which was added to the RRDtool
plugin in version 5.3.0 to fix this issue:
<http://collectd.org/documentation/manpages/collectd.conf.5.shtml#createfilesasync_false_true2>
See also: <https://github.com/collectd/collectd/pull/262>.

Hope this helps :) Best regards,
—octo
-- 
collectd – The system statistics collection daemon
Website: http://collectd.org
Google+: http://collectd.org/+
GitHub:  https://github.com/collectd
Twitter: http://twitter.com/collectd
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20130818/5327eb4f/attachment.pgp>