Benchmarking EBS Volumes in Eucalyptus

Follow

Eucalyptus Versions:  3.2.0 and Greater

Benchmarking EBS Volumes in Eucalyptus

Eucalyptus Elastic Block Store

As part of Eucalyptus EC2 API compatibility with AWS, Eucalyptus supports Elastic Block Store (EBS) volumes and snapshots.  The component responsible for EBS volumes and snapshots is the Storage Controller (SC).  The difference between AWS and Eucalyptus with regards to this feature is that the cloud administrator can determine how the Storage Controller supports providing EBS volumes and snapshots.  Eucalyptus Storage Controller [1] currently supports with the following adaptors [2]:

  • Overlay - the Storage Controller uses iSCSI daemon (tgtd) to support volumes and snapshots.
  • Direct Attached Storage (Just A Bunch Of Disks - JBOD) - the Storage Controller has a JBOD device attached directly to it, using LVM to manage volume and snapshot creation on the JBOD.  iSCSI daemon (tgtd) is used to handle snapshots and volumes.
  • Dell Equallogic (PS 4000, PS 6000 series) SAN (*) - the Storage Controller uses a module to communicate to Dell Equallogic SAN to manage volumes and snapshots.
  • NetApp (FAS 2000, FAS 3000, FAS 6000) SAN (*) - the Storage Controller uses a module to communicate to NetApp 7-mode Filer or Clustered ONTAP to manage volumes and snapshots.
  • EMC VNX SAN (*) - the Storage Controller uses a module to communicate to the EMC VNX SAN to manage volumes and snapshots.

(*) Adaptors with the asterisks denotes that multipathing is supported [3].

With these options, a cloud administrator can use different Storage Controller adapter options per cluster (i.e. availability zone) based upon the hardware that is accessible.  In addition, the cloud administrator can leverage different ways to tune performance based upon which adapter is used within an availability zone's hardware infrastructure.  

Benchmarking Volumes

To get a sense of the performance cloud users will experience when volumes are attached to instances in a given cluster (i.e. availability zone), there are various tools that can be used to benchmark I/O.  This knowledge base will discuss using the tool fio [4], which is used to benchmark and stress testing.  Running benchmark tests give the cloud administrator information to get a better sense what performance tuning can be done on the Node Controller (NC), Storage Controller (SC), network hardware (e.g. enabling jumbo frames on switches and network cards), and SANs - depending upon what adapter is used with the Storage Controller (SC).

In the example below, the Eucalyptus Storage Controller is configured to use the Overlay adapter (tgtd).  Three volumes were attached to an instance running Ubuntu Saucy (13.10), and set up as a RAID0 device using mdadm.  After being formatted to use an XFS filesystem, the device was mounted and fio was used to perform benchmarking.  The steps here follow closely to the steps mentioned in the AWS EC2 documentation discussing benchmark testing for EBS volumes [5].

Steps

  • Install xfsprogs, mdadm and fio on the instance:
ubuntu@172:~$ sudo apt-get install -y xfsprogs mdadm fio
  • Once the instance is launched and the volumes are attached, create a RAID0 device using mdadm:
ubuntu@172:~$ sudo mdadm --create /dev/md0 --level=0 --chunk=64 --raid-devices=3 /dev/vdc /dev/vdd /dev/vde
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
  • Next, make a directory to mount the device, format the device with an XFS filesystem, mount, and change permissions so the non-root user 'ubuntu' can access the device:
ubuntu@172:~$ sudo mkdir -p /media/p_vol0 && sudo mkfs.xfs /dev/md0 && sudo mount -t xfs /dev/md0 /media/p_vol0 && sudo chown ubuntu:ubuntu /media/p_vol0/
meta-data=/dev/md0 isize=256 agcount=16, agsize=245744 blks
= sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=3931904, imaxpct=25
= sunit=16 swidth=48 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
  • Pre-warm the device.  For more information about pre-warming volumes, read the documentation in the AWS EC2 User Guide [6]:
ubuntu@172:~$ sudo dd if=/dev/md0 of=/dev/null
31456896+0 records in
31456896+0 records out
16105930752 bytes (16 GB) copied, 144.203 s, 112 MB/s
  • Confirm that the RAID device has finished syncing by using mdadm:
ubuntu@172:~$ sudo mdadm --query --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Apr 24 01:49:37 2014
Raid Level : raid0
Array Size : 15728448 (15.00 GiB 16.11 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Thu Apr 24 01:49:37 2014
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Chunk Size : 64K
Name : 172:0 (local to host 172)
UUID : 1a8564e8:0539b59b:070011f9:ec65758e
Events : 0
Number Major Minor RaidDevice State
0 253 32 0 active sync /dev/vdc
1 253 48 1 active sync /dev/vdd
2 253 64 2 active sync /dev/vde
  • Finally, use fio to run a benchmark test.  The following example performs 16 KB random write operations:
ubuntu@172:~$ fio --directory=/media/p_vol0 --name p_vol0_fio --direct=1 --rw=randwrite --bs=16k --size=1G --numjobs=16 --time_based --runtime=180 --group_reporting
p_vol0_fio: (g=0): rw=randwrite, bs=16K-16K/16K-16K, ioengine=sync, iodepth=1
...
p_vol0_fio: (g=0): rw=randwrite, bs=16K-16K/16K-16K, ioengine=sync, iodepth=1
2.0.8
Starting 16 processes
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
fio: posix_fallocate fails: No space left on device
p_vol0_fio: Laying out IO file(s) (1 file(s) / 1024MB)
fio: posix_fallocate fails: No space left on device
Jobs: 16 (f=16): [wwwwwwwwwwwwwwww] [100.0% done] [0K/4747K /s] [0 /296 iops] [eta 00m:00s]
p_vol0_fio: (groupid=0, jobs=16): err= 0: pid=6949
write: io=2563.2MB, bw=14572KB/s, iops=910 , runt=180109msec
clat (usec): min=360 , max=4028.3K, avg=17554.31, stdev=113245.06
lat (usec): min=361 , max=4028.3K, avg=17555.38, stdev=113245.05
clat percentiles (usec):
| 1.00th=[ 524], 5.00th=[ 580], 10.00th=[ 620], 20.00th=[ 708],
| 30.00th=[ 804], 40.00th=[ 996], 50.00th=[ 2064], 60.00th=[ 2224],
| 70.00th=[ 2288], 80.00th=[ 2352], 90.00th=[ 2512], 95.00th=[30848],
| 99.00th=[528384], 99.50th=[831488], 99.90th=[1531904], 99.95th=[1826816],
| 99.99th=[3031040]
bw (KB/s) : min= 9, max= 7776, per=8.71%, avg=1268.60, stdev=1952.33
lat (usec) : 500=0.47%, 750=24.32%, 1000=15.40%
lat (msec) : 2=8.85%, 4=44.17%, 10=0.72%, 20=0.80%, 50=1.11%
lat (msec) : 100=1.07%, 250=1.25%, 500=0.72%, 750=0.57%, 1000=0.22%
lat (msec) : 2000=0.30%, >=2000=0.04%
cpu : usr=0.05%, sys=0.36%, ctx=168438, majf=0, minf=375
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=164033/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=2563.2MB, aggrb=14571KB/s, minb=14571KB/s, maxb=14571KB/s, mint=180109msec, maxt=180109msec
Disk stats (read/write):
md0: ios=0/190286, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/63476, aggrmerge=0/4, aggrticks=0/1064474, aggrin_queue=1140690, aggrutil=100.00%
vdc: ios=0/63483, merge=0/4, ticks=0/2770680, in_queue=2885036, util=100.00%
vdd: ios=0/63332, merge=0/5, ticks=0/143764, in_queue=258108, util=100.00%
vde: ios=0/63614, merge=0/3, ticks=0/278980, in_queue=278928, util=74.46%

The 'bw' result shows the average bandwidth achieved by the test.  The 'clat' and 'bw' lines show information about the completion latency and bandwidth respectively. The 'cpu' line shows how much impact the IO load had on the CPU.  For more information about all the details provided by fio, check out the man page for fio [7].  

In summary, benchmarking EBS volumes gives a cloud administrator insight as the performance users may experience on a given availability zone (i.e. Cluster).  We highly recommend a series of tests are performed while the cloud is being tested with tests are being executed.  Based upon the results of these series of tests, information can be gathered to discover bottlenecks in the infrastructure. 

 

References

[1]   Eucalyptus 3.4 Installation Guide - Configuring the Storage Controller
[2]   Eucalyptus Cloud Compatibility Matrix
[3]   Eucalyptus Installation Guide - Configuring Dell Equallogic SAN Multipathing
       Eucalyptus Installation Guide - Configuring NetApp SAN Multipathing
       Eucalyptus Installation Guide - Configuring EMC VNX Multipathing
[4]   Inspecting disk IO performance with fio
[5]   Amazon Elastic Compute Cloud (2014-02-01) - Benchmark Volumes
[6]   Amazon Elastic Compute Cloud (2014-02-01) - Pre-Warm Volumes
[7]   fio(1) - Linux man page

 

 

Have more questions? Submit a request

Comments

Powered by Zendesk