先日のプロジェクトXに大学の学科の先輩が出ていた。もちろん面識はない。そんなとある大学での１年次での話。
１年次の物理実験で放射能を測定する実験があったのだが、その担当の教官の発音が悪く、測定３回したものを考察せよというのが、1000回と聞こえることがあるらしい。まだ１年次というとアホで疑う、逆らうことを知らない学年である。真面目ににみんな夜遅くまで残って、1000回測定して帰ったのである。とある日、その教官が、「自分の実験、みんな1000回測定してくるんだよなぁ」と。（もちろんみんな過去レポも見ている。過去レポも1000回測定された結果が記載されている。）
よく見ると、1000回と言っているときの指は３本出している。「先生３回の測定でいいんですか？」「そうですよ、３回（1000回）でいいと」。長期間潜んだ音声バグを発見した。
1000回測定してデータを出すって、ワークステーションのへのデータ入力が大変なんだ。解析は、友人のプログラムを拝借したが。
閑話休題
IOPS、スループット、平均レイテンシ、標準偏差などを表示する機能があり、複数スレッド、非同期 IO、CPU アフィニティ指定などが使える。
補足として、diskspd-for-linux は Microsoft が参考実装的に公開しているもので、fio のような成熟度はまだ低いですが、Windows 版 DiskSpd とほぼ同じパラメータ体系を維持しているため、クロスプラットフォームの比較には非常に便利。
しかし、コンパイルしてみたらすごい環境依存をする。動かない環境もある。色々調べた結果、簡単にはRHEL8でしか動かない。
開発環境のインストール(RHEL8)
sudo dnf groupinstall -y “Development Tools”
sudo dnf install -y libaio-devel
ダウンロード・コンパイル
git clone git@github.com:microsoft/diskspd-for-linux.git
cd diskspd-for-linux/
make CXXFLAGS=”-O2 -pipe -static-libstdc++ -static-libgcc” LDLIBS=”-laio -lpthread -lrt”
sudo install -m 755 bin/diskspd /usr/local/bin/diskspd
ヘルプはこちら
$ /usr/local/bin/diskspd –help
Usage: diskspd [OPTION…] FILE [FILE…]
Disk I/O benchmarking tool. Specify desired options followed by name(s) of at
least one file or disk to do I/O on. Do not leave spaces between an option and
argument
NOTE: If you are familiar with the Windows version of this tool, note that some
options may behave differently or have slightly different defaults.
-a, –cpu-affinity=CPU_SET By default, threads are affinitized round-robin
across all online cpus in the system. Use this
option to limit the cpus used by providing a cpu
set as an argument. Use comma-delimited groups of
cpus ids to specify a set e.g. “-a 0-3,7” = cpus
0,1,2,3,7
-b, –block-size=BLOCK_SIZE[K|M|G]
Block size in bytes or KiB(K), MiB(M), or GiB(G)
(default=64K)
-B, –base-offset=BASE_OFFSET[K|M|G|b]
Base target offset in bytes or KiB(K), MiB(M),
GiB(G), or blocks(b) from the beginning of the
target (default=0). Must be less than target-size
(-f). i.e. areas of the file outside the interval
[base-offset,target-size) will not be touched.
-c, –create-files=FILE_SIZE[K|M|G|b]
Create files of the specified size in bytes or
KiB(K), MiB(M), GiB(G), or blocks(b).
-d, –duration=DURATION Duration of measurement period in seconds, not
including cooldown or warmup time (default=10)
-D, –iops-std-dev[=INTERVAL] Calculate IOPs standard deviation, and
specify millisecond intervals for bucketing IOPs
data. These are per-thread per-target. You can
specify this without an argument and it will
default to 1000 ms or 1 sec.
-f, –target-size=MAX_FILE_SIZE[K|M|G|b]
Use only the first <arg> bytes or KiB(K), MiB(M),
GiB(G) or blocks(b) of the specified targets, for
example to test only the first sectors of a disk.
Must be greater than base offset (-B). i.e. areas
of the file outside the interval
[base-offset,target-size) will not be touched.
-F, –total-threads=TOTAL_THREADS
Total number of threads. Conflicts with -t, the
option to set the number of threads per file.
-g, –throttle-throughput=THROUGHPUT_PER_MS[K|M|G|b]
Throughput per-thread per-target is throttled to
the given number of bytes, KiB(K), MiB(M), GiB(G)
or blocks(b) per millisecond. NOTE: this option
has varying accuracy depending on number of
threads (-t/-F) and cpu usage.
-L, –latency Measure latency statistics – avg latency of IOPS
per-thread per-target, and standard deviation.
-n, –no-affinity Disable cpu affinity (default and -a).
-o, –overlap=OVERLAP Number of outstanding I/O requests per-thread
per-target. (default=2) Also known as io-depth.
-r, –random-align[=RANDOM_ALIGNMENT[K|M|G|b]]
Random I/O aligned to the specified number of
bytes or KiB(K), MiB(M), GiB(G), or blocks(b).
Overrides -s. Omit the argument to align to block
size by default.
-s, –sequential-stride=[i]STRIDE_SIZE[K|M|G|b]
Sequential stride size, offset between subsequent
I/O operations per-thread in bytes or KiB(K),
MiB(M), GiB(G), or blocks(b). Ignored if -r
specified (default access = sequential, default
stride = block size).By default each thread tracks
its own sequential offset. If the optional
interlocked (i) qualifier is used, a single
interlocked offset is shared between all threads
operating on a given target so that the threads
cooperatively issue a single sequential pattern of
access to the target.
-S, –caching-options=[d|s|h] Modifies caching behavior for targets by
altering the flags passed to open(). By default,
no special flags are specified – i.e. caching is
on. d = O_DIRECT flag; this disables caching for
this file, but the device may still buffer
requests. s = O_SYNC flag; write requests only
return when data has been written to the
underlying device. h = both O_DIRECT and O_SYNC
are used.
-t, –threads-per-target=THREADS_PER_TARGET
Number of threads per target. Conflicts with -F,
which specifies the total number of threads
(default=1).
-T, –thread-stride=THREAD-STRIDE[K|M|G|b]
Stride size between starting offsets of each
thread operating on the same target in bytes or
KiB(K), MiB(M), GiB(G), or blocks(b). (default =
0) The starting offset of a thread = base file
offset + thread number * thread stride. Has no
effect if there is only one thread per target.
-v, –verbose Enable verbose mode – print out details of
operations as they happen.
-w, –write[=WRITE_PERCENTAGE] Percentage of write requests to issue
(default=0, i.e 100% read). The following are
equivalent and result in a 100% read-only
workload: omitting -w, specifying -w with no
percentage, and -w0. IMPORTANT: a write test will
destroy existing data without a warning.
-W, –warmup-time=WARMUP_TIME Duration in seconds to run the test before
results start being recorded (default = 5
seconds).
-x, –io-engine=[k|p] Which io engine to use. k = libaio (kernel aio
interface), p = posix aio (userspace
implementation) default=k
-z, –rand-seed[=RAND_SEED] Set random seed to specified integer value.
With no -z, seed=0. With plain -z, seed is based
on C++’s random_device.
-Z, –io-buffers=[zrs] By default, buffers are shared between reads and
writes, and contain a repeating pattern
(0,1,2…255,0,1).
z = zero the buffers instead. r = fill the buffers
with random data. s = separate the read and write
buffers. z and r conflict.
-?, –help Give this help list
–usage Give a short usage message
Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.
By Nuno Das Neves (t-nudasn at microsoft dot com)
出力結果
# diskspd -f8G -b1M -d5 -o8 -t1 -Sh -L ./testfile
Command Line: diskspd -f8G -b1M -d5 -o8 -t1 -Sh -L ./testfile
System info:
processor count: 1
caching options: fua=0
Input parameters:
job: 1
________
duration: 5s
warm up time: 5s
measuring latency
random seed: 0
total threads: 1
path: ‘./testfile’
size: 8589934592B
using O_DIRECT
using O_SYNC
performing mix test (read/write ratio: 100/0)
block size: 1048576
using sequential I/O (stride: 1048576)
number of outstanding I/O operations: 8)
thread stride size: 0
threads per file: 1
block device: dm-0
device scheduler:
Results for job 1:
test time: 5s
*****************************************************
CPU | Usage | User | Kernel | IO Wait | Idle
——————————————————-
0 | 2.64% | 0.61% | 2.03% | 0.00% | 97.36%
——————————————————-
avg: 2.64% | 0.61% | 2.03% | 0.00% | 97.36%
Total IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat(ms) | LatStdDev | file
——————————————————————————————————-
0 | 5450498048 | 5198 | 1039.60 | 1039.60 | 7.708 | 5.453 | ./testfile (8589934592B)
——————————————————————————————————-
total: 5450498048 | 5198 | 1039.60 | 1039.60 | 7.708 | 5.453
Read IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat(ms) | LatStdDev | file
——————————————————————————————————-
0 | 5450498048 | 5198 | 1039.60 | 1039.60 | 7.708 | 5.453 | ./testfile (8589934592B)
——————————————————————————————————-
total: 5450498048 | 5198 | 1039.60 | 1039.60 | 7.708 | 5.453
Write IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat(ms) | LatStdDev | file
——————————————————————————————————-
0 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | ./testfile (8589934592B)
——————————————————————————————————-
total: 0 | 0 | 0.00 | 0.00 | 0.000 | N/A
%-ile | Read (ms) | Write (ms) | Total (ms)
———————————————-
min | 0.250 | N/A | 0.250
25th | 4.945 | N/A | 4.945
50th | 5.540 | N/A | 5.540
75th | 9.346 | N/A | 9.346
90th | 16.073 | N/A | 16.073
95th | 21.331 | N/A | 21.331
99th | 27.104 | N/A | 27.104
3-nines | 33.257 | N/A | 33.257
4-nines | 43.632 | N/A | 43.632
5-nines | 43.632 | N/A | 43.632
6-nines | 43.632 | N/A | 43.632
7-nines | 43.632 | N/A | 43.632
8-nines | 43.632 | N/A | 43.632
9-nines | 43.632 | N/A | 43.632
max | 43.632 | N/A | 43.632
他の環境で動かせるようにしたい。このためにRHEL8系を作るのはしばし面倒。