I have an Azure Service Fabric cluster with 5 nodes, baked by an Azure Virtual Machine Scale Set. The size of the instances is Standard_D2ds_v4 with an advertised disk speed of max 3200 IOPS. At the VMSS there's a "Premium SSD LRS" data disk attached with the size of 50 GB / 240 Max IOPS / 50 MBps throughput.
Since I need this disk for an EventStoreDB database, I guess 240 IOPS is too slow for production. So I tried another "Premium SSD LRS" data disk with 257 GB size which should reach 2300 Max IOPS / 150 MBps throughput. Still slow, especially for a SSD, but at least better than the first disk.
Before I started the migration of the database to the new disk, I wanted to be sure and measured the speed and realized that the new disk is exactly the same slow as the first disk.
I used the DISKSPD tool as documented at the Microsoft Docs to measure. I got the following results:
PS C:\Users\<user>\Downloads\DiskSpd\amd64> .\diskspd.exe -c10G -w50 -b8K -F4 -r -o128 -W30 -d30 -Sh testfile.dat
Command Line: C:\Users\<user>\Downloads\DiskSpd\amd64\diskspd.exe -c10G -w50 -b8K -F4 -r -o128 -W30 -d30 -Sh testfile.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 30s
cool down time: 0s
random seed: 0
thread pool with 4 threads
number of outstanding I/O operations per thread: 0
path: 'testfile.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 50/50)
block size: 8KiB
using random I/O (alignment: 8KiB)
number of outstanding I/O operations per thread: 128
IO priority: normal
System information:
computer name: prod000000
start time: 2022/06/20 08:03:39 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 4
proc count: 2
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 5.68%| 2.34%| 3.33%| 94.32%
1| 4.17%| 3.23%| 0.94%| 95.83%
-------------------------------------------
avg.| 4.92%| 2.79%| 2.14%| 95.08%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 46063616 | 5623 | 1.46 | 187.41 | testfile.dat (10GiB)
1 | 45998080 | 5615 | 1.46 | 187.14 | testfile.dat (10GiB)
2 | 46170112 | 5636 | 1.47 | 187.84 | testfile.dat (10GiB)
3 | 46120960 | 5630 | 1.47 | 187.64 | testfile.dat (10GiB)
------------------------------------------------------------------------------
total: 184352768 | 22504 | 5.86 | 750.03
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 23109632 | 2821 | 0.73 | 94.02 | testfile.dat (10GiB)
1 | 22732800 | 2775 | 0.72 | 92.49 | testfile.dat (10GiB)
2 | 22986752 | 2806 | 0.73 | 93.52 | testfile.dat (10GiB)
3 | 23117824 | 2822 | 0.73 | 94.05 | testfile.dat (10GiB)
------------------------------------------------------------------------------
total: 91947008 | 11224 | 2.92 | 374.08
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 22953984 | 2802 | 0.73 | 93.39 | testfile.dat (10GiB)
1 | 23265280 | 2840 | 0.74 | 94.65 | testfile.dat (10GiB)
2 | 23183360 | 2830 | 0.74 | 94.32 | testfile.dat (10GiB)
3 | 23003136 | 2808 | 0.73 | 93.59 | testfile.dat (10GiB)
------------------------------------------------------------------------------
total: 92405760 | 11280 | 2.94 | 375.95
PS C:\Users\<user>\Downloads\DiskSpd\amd64> .\diskspd.exe -c10G -w50 -b8K -F4 -r -o128 -W30 -d30 -Sh D:\testfile.da
t
Command Line: C:\Users\<user>\Downloads\DiskSpd\amd64\diskspd.exe -c10G -w50 -b8K -F4 -r -o128 -W30 -d30 -Sh D:\testfile.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 30s
cool down time: 0s
random seed: 0
thread pool with 4 threads
number of outstanding I/O operations per thread: 0
path: 'D:\testfile.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 50/50)
block size: 8KiB
using random I/O (alignment: 8KiB)
number of outstanding I/O operations per thread: 128
IO priority: normal
System information:
computer name: prod000000
start time: 2022/06/20 08:07:46 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.01s
thread count: 4
proc count: 2
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 12.39%| 2.19%| 10.20%| 87.61%
1| 12.18%| 5.10%| 7.08%| 87.82%
-------------------------------------------
avg.| 12.29%| 3.64%| 8.64%| 87.71%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 629145600 | 76800 | 19.99 | 2558.97 | D:\testfile.dat (10GiB)
1 | 1155129344 | 141007 | 36.71 | 4698.34 | D:\testfile.dat (10GiB)
2 | 634142720 | 77410 | 20.15 | 2579.29 | D:\testfile.dat (10GiB)
3 | 1145102336 | 139783 | 36.39 | 4657.56 | D:\testfile.dat (10GiB)
------------------------------------------------------------------------------
total: 3563520000 | 435000 | 113.24 | 14494.16
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 315432960 | 38505 | 10.02 | 1282.98 | D:\testfile.dat (10GiB)
1 | 579280896 | 70713 | 18.41 | 2356.15 | D:\testfile.dat (10GiB)
2 | 317087744 | 38707 | 10.08 | 1289.71 | D:\testfile.dat (10GiB)
3 | 574537728 | 70134 | 18.26 | 2336.86 | D:\testfile.dat (10GiB)
------------------------------------------------------------------------------
total: 1786339328 | 218059 | 56.76 | 7265.71
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 313712640 | 38295 | 9.97 | 1275.99 | D:\testfile.dat (10GiB)
1 | 575848448 | 70294 | 18.30 | 2342.19 | D:\testfile.dat (10GiB)
2 | 317054976 | 38703 | 10.07 | 1289.58 | D:\testfile.dat (10GiB)
3 | 570564608 | 69649 | 18.13 | 2320.70 | D:\testfile.dat (10GiB)
------------------------------------------------------------------------------
total: 1777180672 | 216941 | 56.47 | 7228.45
PS C:\Users\<user>\Downloads\DiskSpd\amd64> .\diskspd.exe -c10G -w50 -b8K -F4 -r -o128 -W30 -d30 -Sh S:\testfile.da
t
Command Line: C:\Users\<user>\Downloads\DiskSpd\amd64\diskspd.exe -c10G -w50 -b8K -F4 -r -o128 -W30 -d30 -Sh S:\testfile.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 30s
cool down time: 0s
random seed: 0
thread pool with 4 threads
number of outstanding I/O operations per thread: 0
path: 'S:\testfile.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 50/50)
block size: 8KiB
using random I/O (alignment: 8KiB)
number of outstanding I/O operations per thread: 128
IO priority: normal
System information:
computer name: prod000000
start time: 2022/06/20 08:11:46 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 4
proc count: 2
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 9.48%| 1.30%| 8.18%| 90.52%
1| 7.14%| 4.64%| 2.50%| 92.86%
-------------------------------------------
avg.| 8.31%| 2.97%| 5.34%| 91.69%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 215162880 | 26265 | 6.84 | 875.46 | S:\testfile.dat (10GiB)
1 | 214958080 | 26240 | 6.83 | 874.63 | S:\testfile.dat (10GiB)
2 | 215547904 | 26312 | 6.85 | 877.03 | S:\testfile.dat (10GiB)
3 | 214466560 | 26180 | 6.82 | 872.63 | S:\testfile.dat (10GiB)
------------------------------------------------------------------------------
total: 860135424 | 104997 | 27.34 | 3499.75
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 107978752 | 13181 | 3.43 | 439.35 | S:\testfile.dat (10GiB)
1 | 107872256 | 13168 | 3.43 | 438.91 | S:\testfile.dat (10GiB)
2 | 107470848 | 13119 | 3.42 | 437.28 | S:\testfile.dat (10GiB)
3 | 107282432 | 13096 | 3.41 | 436.51 | S:\testfile.dat (10GiB)
------------------------------------------------------------------------------
total: 430604288 | 52564 | 13.69 | 1752.06
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 107184128 | 13084 | 3.41 | 436.11 | S:\testfile.dat (10GiB)
1 | 107085824 | 13072 | 3.40 | 435.71 | S:\testfile.dat (10GiB)
2 | 108077056 | 13193 | 3.44 | 439.75 | S:\testfile.dat (10GiB)
3 | 107184128 | 13084 | 3.41 | 436.11 | S:\testfile.dat (10GiB)
------------------------------------------------------------------------------
total: 429531136 | 52433 | 13.65 | 1747.69
PS C:\Users\<user>\Downloads\DiskSpd\amd64> .\diskspd.exe -c10G -w50 -b8K -F4 -r -o128 -W30 -d30 -Sh T:\testfile.da
t
Command Line: C:\Users\<user>\Downloads\DiskSpd\amd64\diskspd.exe -c10G -w50 -b8K -F4 -r -o128 -W30 -d30 -Sh T:\testfile.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 30s
cool down time: 0s
random seed: 0
thread pool with 4 threads
number of outstanding I/O operations per thread: 0
path: 'T:\testfile.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 50/50)
block size: 8KiB
using random I/O (alignment: 8KiB)
number of outstanding I/O operations per thread: 128
IO priority: normal
System information:
computer name: prod000000
start time: 2022/06/20 08:13:01 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 4
proc count: 2
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 9.17%| 1.51%| 7.66%| 90.83%
1| 3.39%| 1.93%| 1.46%| 96.61%
-------------------------------------------
avg.| 6.28%| 1.72%| 4.56%| 93.72%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 214761472 | 26216 | 6.83 | 873.79 | T:\testfile.dat (10GiB)
1 | 215842816 | 26348 | 6.86 | 878.18 | T:\testfile.dat (10GiB)
2 | 214679552 | 26206 | 6.82 | 873.45 | T:\testfile.dat (10GiB)
3 | 215810048 | 26344 | 6.86 | 878.05 | T:\testfile.dat (10GiB)
------------------------------------------------------------------------------
total: 861093888 | 105114 | 27.37 | 3503.47
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 107831296 | 13163 | 3.43 | 438.73 | T:\testfile.dat (10GiB)
1 | 108183552 | 13206 | 3.44 | 440.16 | T:\testfile.dat (10GiB)
2 | 106881024 | 13047 | 3.40 | 434.86 | T:\testfile.dat (10GiB)
3 | 108150784 | 13202 | 3.44 | 440.03 | T:\testfile.dat (10GiB)
------------------------------------------------------------------------------
total: 431046656 | 52618 | 13.70 | 1753.77
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 106930176 | 13053 | 3.40 | 435.06 | T:\testfile.dat (10GiB)
1 | 107659264 | 13142 | 3.42 | 438.03 | T:\testfile.dat (10GiB)
2 | 107798528 | 13159 | 3.43 | 438.59 | T:\testfile.dat (10GiB)
3 | 107659264 | 13142 | 3.42 | 438.03 | T:\testfile.dat (10GiB)
------------------------------------------------------------------------------
total: 430047232 | 52496 | 13.67 | 1749.70
I ran the test on all disks for comparison:
- C:\ -> OS Disk
- D:\ -> Ephemeral Disk (not usable for a database, as the data can be lost during certain maintenance work)
- S:\ -> the 50 GB data disk
- S:\ -> the 257 GB data disk
As you can see, I'm only achieving a fraction of the IOPS and throughput of the specified speed, and the two data disks are equally slow. Have I configured something wrong?
EDIT: It looks like the IOPS are ok. I had assumed that reading and writing happened one after the other and accordingly the IOPS are at 1750. The total measured 3500 IOPS are even higher than the 2800 stated as maximum. EDIT 2: the 3500 IOPS are probably related to the bursting on managed Premium SSDs.
However, it doesn't look so good with the throughput, where the 27.3 MBps are definitely lower than the stated 50 MBps resp. 150 MBps. EDIT 2: with bursting I should reach 170 MBps, without 150 MBps.
The question why the two drives are equally fast also remains.
