Storage and Networking Performance
Beware: This post will be of little interest to those
who are primarily in it for the physical side of
building. Instead, this update will be about the performance
and software side of things. So, lots of text, lots of
numbers.
These results are still somewhat preliminary since I'm not
yet 100% sure if the hardware config will remain like this
for an extended period of time (I really want to put another
12 GB of RAM in there, for example, and am considering
adding some SSD goodness to my ZFS pools), nor am I
necessarily done with tuning software parameters, but it
should give some idea of what performance I'm currently
getting.
As you may recall from my previous update, I'm running three
VMs on this machine, two of which are pretty much always on
(the media VM and my personal VM), and the third of which is
only active when I'm pulling a backup of my dad's work
machine (apollo-business).
NOTE: I know there's lots of text and stuff in my
screenshots and it may be a bit difficult to read. Click
on any image to get the full-res version for improved
legibility.
The storage setup has been revised somewhat since the last
update. I now have a mirrored ZFS pool in ZEUS for backing
up my dad's business data (so, in total his data is on six
HDDs, including the one in his work machine). His data is
pulled onto the apollo-business VM from his work machine,
and then pulled onto ZEUS. The fact that neither the
business VM nor ZEUS are online 24/7 (ZEUS is turned off
physically most of the time) should provide some decent
protection against most malheurs, the only thing I still
need to implement is a proper off-site backup plan (which
I will definitely do, in case of unforeseen disasters,
break-ins/theft and so on).
(click image for full res)
The Plan
For convenience's sake, I was planning on using NFS for
sharing data between the server and its various clients
on our network. Unfortunately, I was getting some rather
disappointing benchmarking results initially, with only ~60
MB/s to ~70 MB/s transfer speeds between machines.
Tools
I'm not really a storage benchmarking expert, and at the
moment I definitely don't have the time to become one, so
for benchmarking my storage I've used dd for the time
being. It's easy to use and is pretty much standard for
every Linux install. I thought about using other storage
benchmarks like Bonnie++ and FIO, and at some point I might
still do that, but for the time being dd will suffice for my
purposes.
For those not familiar with this: /dev/zero basically
serves as a data source for lots of zeroes, /dev/null is a
sink into which you can write data without it being written
to disk. So, if you want to do writing benchmarks to your
storage, you can grab data from /dev/zero without needing to
worry about a bottleneck on your data source side, and
/dev/null is the equivalent when you wish to do reading
benchmarks. To demonstrate this, I did a quick test below
directly from /dev/zero into /dev/null.
Basically. It's a bit of a simplification, but I hope it's
somewhat understandable.
Baseline
Before doing storage benchmarks across the network, we
should of course get a baseline for both the storage setup
itself as well as the network.
The base pipe from /dev/zero into /dev/null transfers has a
transfer speed of ~9 GB/s. Nothing unexpected, but it's a
quick test to do and I was curious about this:
(click image for full res)
For measuring this I used iperf, here's a screencap from one
of my test runs. The machine it's running on was my personal
VM.
Top to bottom:
- my dad's Windows 7 machine
- APOLLO host (Arch Linux)
- HELIOS (also Windows 7 for the time being, sadly)
- ZEUS (Arch Linux)
- My Laptop via WiFi (Arch Linux)
- APOLLO business VM (Arch Linux)
- APOLLO media VM
The bottom two results aren't really representative of
typical performance, usually it's ~920 Mbit/s to ~940
Mbit/s, But as with any setup, outliers happen.
(click image for full res)
The networking performance is where I hit my first hickup.
I failed to specify to the VM which networking driver it was
supposed to use, and the default one does not exactly have
stellar performance. It was an easy fix though, and with the
new settings I now get pretty much the same networking
performance across all my machines (except the Windows ones,
those are stuck at ~500 Mbit/s for some reason as you can
see above, but that's not hugely important to me at the
moment TBH).
This is representative of what I can get most of the time:
(click image for full res)
I had a similar issue with the storage subsystem at first,
the default parameters for caching were not very conducive
to high performance and resulted in some pretty bad results:
(click image for full res)
Once I fixed that though, much better, and sufficient to
saturate a gigabit networking connection.
(click image for full res)
Networking Benchmark Results
Initially, I got only around 60 MB/s for NFS, after that the
next plateau was somewhere between 75 MB/s and 80 MB/s, and
lastly, this is the current situation. I must say I find the
results to be slightly... peculiar. Pretty much everything
I've ever read says that NFS should offer better performance
than CIFS, and yet, for some reason, in many cases that was
not the result I got.
I'm not yet sure if I'll be going with NFS or CIFS in the
end to be honest. On one hand, CIFS does give my better
performance for the most part, but I have found NFS more
convenient to configure and use, and NFS' performance at
this point is decent enough for most of my purposes.
In general, I find the NFS results just rather weird
TBH. But they have been reproducible over different runs on
several days, so for the time being I'll accept them as what
I can get.
Anyway, behold the mother of all graphics!
(click image for full res)
FTP
As an alternative, I've also tried FTP , but results were
not really very satisfying. This is just a screenshot from
one test run, but it is representative of the various other
test runs I did:
(click image for full res)
ZFS Compression
Also, for those curious about ZFS' compression (which was
usually disabled in the above tests because zeroes are very
compressible and would therefore skew the benchmarks), I did
a quick test to compare writing zeroes to a ZFS pool with
and without compression.
This is CPU utilization without compression (the grey bars
are CPU time spent waiting for I/O, not actual work the CPU
is doing):
(click image for full res)
And this was the write speed for that specific test run:
(click image for full res)
With lz4 compression enabled, the CPU does quite a bit more
work, as expected (though it still seems that you don't
really need a very powerful CPU to make use of this):
(click image for full res)
And the write speed goes up almost to a gigabyte per second,
pretty neat if you ask me.
(click image for full res)
Side note: ZFS' lz4 compression is allegedly smart enough
not to try to compress incompressible data, such as media
files which are already compressed, which should prevent
such writes from being slowed down. Very nice IMHO.
That's it for today. What's still left to do at this point
is installing some sound-dampening materials (the rig is a
bit on the loud side, even despite being in its own room),
and possibly upgrading to more RAM, the rest will probably
stay like this for a while. If I really do upgrade to more
RAM, I'll adjust the VMs accordingly and run the tests
again, just to see if that really makes a difference. So far
I have been unable to get better performance from my ZFS
pools by allocating more RAM, or even running benches
directly on the host machine with the full 12 GB RAM and
eight cores/sixteen threads.
Cheers,
-aw
