Performance¶
Environment¶
Cluster Information
Instance | Nodes | CPU | Memory | Storage | Network | Description |
Master | 3 | 32 | 32 GB | 120 GB SSD | 10 Gb/s | |
MetaNode | 10 | 32 | 32 GB | 16 x 1TB SSD | 10 Gb/s | hybrid deployment |
DataNode | 10 | 32 | 32 GB | 16 x 1TB SSD | 10 Gb/s | hybrid deployment |
Volume Setup
#!/bin/bash
# create volume
curl "http://${MASTER}/admin/createVol?name=intest&owner=cfs&capacity=300000000&mpCount=10"
# increase 1500 data partitions
curl "http://${MASTER}/dataPartition/create?name=intest&count=1500"
- Initial number of Meta Partitions:
10
- Number of Meta Replication:
3
- Initial number of Data Partitions:
1510
- Number of Data Replication:
3
Small File Performance and Scalability¶
Small file operation performance and scalability benchmark test by mdtest.
Setup
#!/bin/bash
set -e
TARGET_PATH="/mnt/test/mdtest" # mount point of ChubaoFS volume
for FILE_SIZE in 1024 2048 4096 8192 16384 32768 65536 131072 # file size
do
mpirun --allow-run-as-root -np 512 --hostfile hfile64 mdtest -n 1000 -w $i -e $FILE_SIZE -y -u -i 3 -N 1 -F -R -d $TARGET_PATH;
done
Benchmark
File Size (KB) | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
Creation (TPS) | 70383 | 70383 | 73738 | 74617 | 69479 | 67435 | 47540 | 27147 |
Read (TPS) | 108600 | 118193 | 118346 | 122975 | 116374 | 110795 | 90462 | 62082 |
Removal (TPS) | 87648 | 84651 | 83532 | 79279 | 85498 | 86523 | 80946 | 84441 |
Stat (TPS) | 231961 | 263270 | 264207 | 252309 | 240244 | 244906 | 273576 | 242930 |
IO Performance and Scalability¶
IO Performance and benchmark scalability test by fio.
Note: Multiple clients mount the same volume. And the process refers to the fio process.
1. Sequential Read¶
Setup
#!/bin/bash
fio -directory={} \
-ioengine=psync \
-rw=read \ # sequential read
-bs=128k \ # block size
-direct=1 \ # enable direct IO
-group_reporting=1 \
-fallocate=none \
-time_based=1 \
-runtime=120 \
-name=test_file_c{} \
-numjobs={} \
-nrfiles=1 \
-size=10G
Bandwidth
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 148.000 | 626.000 | 1129.000 | 1130.000 |
2 Clients | 284.000 | 1241.000 | 2258.000 | 2260.000 |
4 Clients | 619.000 | 2640.000 | 4517.000 | 4515.000 |
8 Clients | 1193.000 | 4994.000 | 9006.000 | 9034.000 |
IOPS
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 1180.000 | 5007.000 | 9031.000 | 9040.000 |
2 Clients | 2275.000 | 9924.000 | 18062.000 | 18081.000 |
4 Clients | 4954.000 | 21117.000 | 36129.000 | 36112.000 |
8 Clients | 9531.000 | 39954.000 | 72048.000 | 72264.000 |
Latency
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 842.200 | 794.340 | 1767.310 | 7074.550 |
2 Clients | 874.255 | 801.690 | 1767.370 | 7071.715 |
4 Clients | 812.363 | 760.702 | 1767.710 | 7077.065 |
8 Clients | 837.707 | 799.851 | 1772.620 | 7076.967 |
2. Sequential Write¶
Setup
#!/bin/bash
fio -directory={} \
-ioengine=psync \
-rw=write \ # sequential write
-bs=128k \ # block size
-direct=1 \ # enable direct IO
-group_reporting=1 \
-fallocate=none \
-name=test_file_c{} \
-numjobs={} \
-nrfiles=1 \
-size=10G
Bandwidth
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 52.200 | 226.000 | 956.000 | 1126.000 |
2 Clients | 104.500 | 473.000 | 1763.000 | 2252.000 |
4 Clients | 225.300 | 1015.000 | 2652.000 | 3472.000 |
8 Clients | 480.600 | 1753.000 | 3235.000 | 3608.000 |
IOPS
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 417 | 1805 | 7651 | 9004 |
2 Clients | 835 | 3779 | 14103 | 18014 |
4 Clients | 1801 | 8127 | 21216 | 27777 |
8 Clients | 3841 | 14016 | 25890 | 28860 |
Latency
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 2385.400 | 2190.210 | 2052.360 | 7081.320 |
2 Clients | 2383.610 | 2081.850 | 2233.790 | 7079.450 |
4 Clients | 2216.305 | 1947.688 | 2946.017 | 8842.903 |
8 Clients | 2073.921 | 2256.120 | 4787.496 | 17002.425 |
3. Random Read¶
Setup
#!/bin/bash
fio -directory={} \
-ioengine=psync \
-rw=randread \ # random read
-bs=4k \ # block size
-direct=1 \ # enable direct IO
-group_reporting=1 \
-fallocate=none \
-time_based=1 \
-runtime=120 \
-name=test_file_c{} \
-numjobs={} \
-nrfiles=1 \
-size=10G
Bandwidth
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 6.412 | 39.100 | 216.000 | 534.000 |
2 Clients | 14.525 | 88.100 | 409.000 | 1002.000 |
4 Clients | 33.242 | 200.200 | 705.000 | 1693.000 |
8 Clients | 59.480 | 328.300 | 940.000 | 2369.000 |
IOPS
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 1641 | 10240 | 56524.800 | 140288 |
2 Clients | 3718 | 23142.4 | 107212.8 | 263168 |
4 Clients | 8508 | 52428.8 | 184627.2 | 443392 |
8 Clients | 15222 | 85072.8 | 246681.6 | 621056 |
Latency
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 603.580 | 395.420 | 287.510 | 466.320 |
2 Clients | 532.840 | 351.815 | 303.460 | 497.100 |
4 Clients | 469.025 | 317.140 | 355.105 | 588.847 |
8 Clients | 524.709 | 382.862 | 530.811 | 841.985 |
4. Random Write¶
Setup
#!/bin/bash
fio -directory={} \
-ioengine=psync \
-rw=randwrite \ # random write
-bs=4k \ # block size
-direct=1 \ # enable direct IO
-group_reporting=1 \
-fallocate=none \
-time_based=1 \
-runtime=120 \
-name=test_file_c{} \
-numjobs={} \
-nrfiles=1 \
-size=10G
Bandwidth
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 3.620 | 17.500 | 118.000 | 318.000 |
2 Clients | 7.540 | 44.800 | 230.000 | 476.000 |
4 Clients | 16.245 | 107.700 | 397.900 | 636.000 |
8 Clients | 39.274 | 208.100 | 487.100 | 787.100 |
IOPS
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 926.000 | 4476.000 | 31027.200 | 83251.200 |
2 Clients | 1929.000 | 11473.000 | 60313.600 | 124620.800 |
4 Clients | 4156.000 | 27800.000 | 104243.200 | 167014.400 |
8 Clients | 10050.000 | 53250.000 | 127692.800 | 206745.600 |
Latency
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 1073.150 | 887.570 | 523.820 | 784.030 |
2 Clients | 1030.010 | 691.530 | 539.525 | 1042.685 |
4 Clients | 955.972 | 575.183 | 618.445 | 1552.205 |
8 Clients | 789.883 | 598.393 | 1016.185 | 2506.424 |
Metadata Performance and Scalability¶
Metadata performance and scalability benchmark test by mdtest.
Setup
#!/bin/bash
TEST_PATH=/mnt/cfs/mdtest # mount point of ChubaoFS volume
for CLIENTS in 1 2 4 8 # number of clients
do
mpirun --allow-run-as-root -np $CLIENTS --hostfile hfile01 mdtest -n 5000 -u -z 2 -i 3 -d $TEST_PATH;
done
Dir Creation
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 448.618 | 2421.001 | 14597.97 | 43055.15 |
2 Clients | 956.947 | 5917.576 | 28930.431 | 72388.765 |
4 Clients | 2027.02 | 13213.403 | 54449.056 | 104771.356 |
8 Clients | 4643.755 | 27416.904 | 89641.301 | 119542.62 |
Dir Removal
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 399.779 | 2118.005 | 12351.635 | 34903.672 |
2 Clients | 833.353 | 5176.812 | 24471.674 | 50242.973 |
4 Clients | 1853.617 | 11462.927 | 46413.313 | 91128.059 |
8 Clients | 4441.435 | 24133.617 | 74401.336 | 115013.557 |
Dir Stat
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 283232.761 | 1215309.524 | 4231088.104 | 12579177.02 |
2 Clients | 572834.143 | 2169669.058 | 8362749.217 | 18120970.71 |
4 Clients | 1263474.549 | 3333746.786 | 10160929.29 | 31874265.88 |
8 Clients | 2258670.069 | 8715752.83 | 22524794.98 | 77533648.04 |
File Creation
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 448.888 | 2400.803 | 13638.072 | 27785.947 |
2 Clients | 925.68 | 5664.166 | 25889.163 | 50434.484 |
4 Clients | 2001.137 | 12986.968 | 50330.952 | 91387.825 |
8 Clients | 4479.831 | 25933.437 | 86667.966 | 112746.199 |
File Removal
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 605.143 | 3678.138 | 18631.342 | 47035.912 |
2 Clients | 1301.151 | 8365.667 | 34005.25 | 64860.041 |
4 Clients | 3032.683 | 14017.426 | 50938.926 | 80692.761 |
8 Clients | 7170.386 | 32056.959 | 68761.908 | 88357.563 |
Tree Creation
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 305.778 | 229.562 | 86.299 | 23.917 |
2 Clients | 161.31 | 211.119 | 76.301 | 24.439 |
4 Clients | 260.562 | 223.153 | 81.209 | 23.867 |
8 Clients | 350.038 | 220.744 | 81.621 | 17.144 |
Tree Removal
1 Process | 4 Processes | 16 Processes | 64 Processes | |
1 Client | 137.462 | 70.881 | 31.235 | 7.057 |
2 Clients | 217.026 | 113.36 | 23.971 | 7.128 |
4 Clients | 231.622 | 113.539 | 30.626 | 7.299 |
8 Clients | 185.156 | 83.923 | 20.607 | 5.515 |
Integrity¶
- Linux Test Project / fs
Workload¶
Database backup
Java application logs
Code git repo
Database systems
MyRocks, MySQL Innodb, HBase,
Scalability¶
- Volume Scalability: tens to millions of cfs volumes
- Metadata Scalability: a big volume with billions of files/directories