安装测试工具

1、安装依赖程序

2、创建软件包目录

3、安装mpi

# cd tools
# curl -O https://download.open-mpi.org/release/open-mpi/v1.10/openmpi-1.10.7.tar.gz
# ./configure --prefix=/usr/local/openmpi/
# make
# make install

4、添加环境变量

在.bashrc文件中添加以下三行内容

# vim /root/.bashrc
export PATH=$PATH:/usr/local/openmpi/bin/:/usr/local/ior/bin/
export LD_LIBRARY_PATH=/usr/local/openmpi/lib:${LD_LIBRARY_PATH}
export MPI_CC=mpicc 
# source /root/.bashrc

5、安装IOR

# cd tools/
# yum -y install git automake
# git clone https://github.com/chaos/ior.git
# cd ior
# ./bootstrap
# ./configure --prefix=/usr/local/ior/
# make
# make install

6、安装mdtest

# cd tools/
# mkdir mdtest
# wget https://nchc.dl.sourceforge.net/project/mdtest/mdtest%20latest/mdtest-1.9.3/mdtest-1.9.3.tgz
# cd mdtest
# tar xf mdtest-1.9.3.tgz
# make

使用IOR和Mdtest进行测试

环境

挂载CephFS

以下操作需要在所有的客户端节点完成

1、在客户端节点安装ceph-fuse

# yum -y install ceph-fuse

2、创建ceph文件保存目录

# mkdir /etc/ceph

3、拷贝服务端的ceph的keyring文件

# scp 10.0.98.21:/etc/ceph/* /etc/ceph/

4、创建挂载目录

5、挂载cephfs

# ceph-fuse -m 10.0.98.21:6789/ /mnt/fs2/
# ceph-fuse -m 10.0.98.21:6789/ /mnt/fs3/
# ceph-fuse -m 10.0.98.21:6789/ /mnt/fs4/

6、在挂载目录下创建测试目录

# mkdir /mnt/fs{1,2,3,4}/iortest

1、安装ceph-fuse

ansible clients -m yum -a ‘name=ceph-fuse state=installed’

2、创建ceph文件目录

ansible clients -m file -a ‘name=/etc/ceph state=directory’

3、拷贝ceph文件

ansible clients -m copy -a ‘src=/etc/ceph/ceph.client.admin.keyring dest=/etc/ceph/ceph.client.admin.keyring’

4、创建挂载目录

ansible clients -m shell -a ‘mkdir -p /mnt/fs{1,2,3,4}’

5、挂载cephfs

ansible clients -m shell -a ‘ceph-fuse -m MDSIP:6789/ /mnt/fs1’
ansible clients -m shell -a ‘ceph-fuse -m MDSIP:6789/ /mnt/fs2’
ansible clients -m shell -a ‘ceph-fuse -m MDSIP:6789/ /mnt/fs3’
ansible clients -m shell -a ‘ceph-fuse -m MDSIP:6789/ /mnt/fs4’

6、在挂载目录下创建测试目录

ansible clients -m shell -a ‘mkdir /mnt/fs{1,2,3,4}/iortest’

创建测试脚本

ior的测试通过scripts完成，以下是几个测试脚本示例，实际测试过程中可以根据测试环境进行修改

1、单个客户端单个进程

2、单个客户端4个进程

# cat test/iorconf-14 
===============> start script <===============
IOR START
  api=POSIX
  hintsFileName=hintsFile
  testFile=/mnt/fs1/iortest/f14-1@/mnt/fs2/iortest/f14-2@/mnt/fs3/iortest/f14-3@/mnt/fs4/iortest/f14-4
  repetitions=1
  readFile=1
  writeFile=1
  filePerProc=1
  segmentCount=1
  blockSize=32g
  transferSize=1024k
  collective=0
===============> stop script <===============

3、5个客户端4个进程

# cat test/iorconf-54
===============> start script <===============
IOR START
  api=POSIX
  hintsFileName=hintsFile
  testFile=/mnt/fs1/iortest/f54-1@/mnt/fs2/iortest/f54-2@/mnt/fs3/iortest/f54-3@/mnt/fs4/iortest/f54-4@/mnt/fs1/iortest/f54-5@/mnt/fs2/iortest/f54-6@/mnt/fs3/iortest/f54-7@/mnt/fs4/iortest/f54-8@/mnt/fs1/iortest/f54-9@/mnt/fs2/iortest/f54-10@/mnt/fs3/iortest/f54-11@/mnt/fs4/iortest/f54-12@/mnt/fs1/iortest/f54-13@/mnt/fs2/iortest/f54-14@/mnt/fs3/iortest/f54-15@/mnt/fs4/iortest/f54-16@/mnt/fs1/iortest/f54-17@/mnt/fs2/iortest/f54-18@/mnt/fs3/iortest/f54-19@/mnt/fs4/iortest/f54-20
  repetitions=1
  readFile=1
  writeFile=1
  filePerProc=1
  segmentCount=1
  blockSize=32g
  transferSize=1024k
  collective=0
IOR STOP
===============> stop script <===============

测试命令

IOR测试IOR测试

1、单个客户端单个进程

# ior -f test/iorconf-11

2、单个客户端4个进程

# mpirun --allow-run-as-root -np 4 ior -f test/iorconf-14

3、5个客户端4个进程

# mpirun --allow-run-as-root --mca plm_rsh_no_tree_spawn 1 -npernode 4 -hostfile test/hosts ior -f test/iorconf-54

# cat test/hosts 
test1 slots=4
test2 slots=4
test3 slots=4
test4 slots=4
test5 slots=4

Mdtest测试

1、单个客户端单个进程

# ./mdtest -F -L -z 4 -b 2 -I 1562 -u -d /mnt/fs1/mdtest/

2、单个客户端4个进程

3、5个客户端4个进程