Tmp

- - - 紀錄於機房三台server設定流程 - - -

Server HostName CAKE配置 對內(mgmt) 對外(web)
No.30 c1 Center+Node 192.168.10.10 140.92.25.126
No.29 n1 Node 192.168.10.11 140.92.25.238
No.32 n2 Node 192.168.10.12 140.92.25.229

 

1. Install EPEL

touch /etc/yum.repos.d/Gluster.repo
[glusterfs-epel]
name=GlusterFS is a clustered file-system capable of scaling to several petabytes.
baseurl=http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-$releasever/$basearch/
enabled=1
skip_if_unavailable=1
gpgcheck=1
gpgkey=http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/pub.key

[glusterfs-noarch-epel]
name=GlusterFS is a clustered file-system capable of scaling to several petabytes.
baseurl=http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-$releasever/noarch
enabled=1
skip_if_unavailable=1
gpgcheck=1
gpgkey=http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/pub.key

[glusterfs-source-epel]
name=GlusterFS is a clustered file-system capable of scaling to several petabytes. - Source
baseurl=http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-$releasever/SRPMS
enabled=0
skip_if_unavailable=1
gpgcheck=1
gpgkey=http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/pub.key

 
 

2. Install glusterfs-server

yum install gdisk glusterfs{,-server,-fuse,-geo-replication} -y

 
 

3. clean Gluster.repo

rm /etc/yum.repos.d/Gluster.repo
yum clean all

 
 

4. glusterfs!

CAKE storage pool
  • center+node
    • /dev/vg_livecd/lv_mnt_storage
  • node
    • /dev/vg_livecd/lv_backup
CAKE mount point
  • for all
    • /mnt/storage

 


停止CAKE服務
- service cakeweb stop (center only)
- service caked stop
- service starterd stop (center only)
- service nfs stop (center only)
 
卸載mount point(for node)
- umount /mnt/storage
 
啟動GlusterFS服務 (for all)
- /etc/init.d/glusterd start or
- service glusterd start
 
加入trusted storage pools
- gluster peer probe n1
- gluster peer probe n2
 
mount the CAKE storage as a Gluster "brick" (center)
- mkdir /mnt/gfbrick
- mount /dev/vg_livecd/lv_mnt_storage /mnt/gfbrick/
- echo "/dev/vg_livecd/lv_mnt_storage /mnt/gfbrick ext4 defaults 1 2" » /etc/fstab (暫時省略)
- mkdir /mnt/gfbrick/brick1
 
mount the CAKE storage as a Gluster "brick" (node)
- mkdir /mnt/gfbrick
- mount /dev/vg_livecd/lv_backup /mnt/gfbrick/
- echo "/dev/vg_livecd/lv_backup /mnt/gfbrick ext4 defaults 1 2" » /etc/fstab (暫時省略)
- mkdir /mnt/gfbrick/brick1
 
 
建立GlusterFS DHT volume
- gluster volume create dht-vol c1:/mnt/gfbrick/brick1 n1:/mnt/gfbrick/brick1 n2:/mnt/gfbrick/brick1
- gluster volume start dht-vol
 
備份資料到Gluster volume (才能以DHT or Striped方式分散資料)
- mkdir /mnt/storage2
- mount -t glusterfs c1:/dht-vol /mnt/storage2/
- rsync -ravP /mnt/storage/* /mnt/storage2/
 
umout previous Gluster volume
- umount /mnt/storage2/ (only for center)
- umount /mnt/storage/ (for all)

mount Gluster volume (for all)
- mount -t glusterfs c1:/dht-vol /mnt/storage/
 
重啟服務
- service caked restart
- service cakeweb restart (center only)

先不要開starterd, 因為mount選項還沒新增DFS mount option

 
End of story!
 
 


 
備註:
暫時不要重啟c1, n1, n2
如果有需要也必須做以下處理
n1, n2

mount 192.168.10.10:/mnt/storage /mnt/storage

after reboot

 
 
 
 

iometer it!

 
 
 

 
 


附錄一 - experimental results

iometer

Link » GlusterFS on CAKE

 
 

dd

查cache (optional)
- free -m
 
清cache
- echo 2 > /proc/sys/vm/drop_caches
 
- cd /mnt/storage
 
測試1
- dd if=/dev/zero of=./hello bs=8192 count=3932160
- dd if=/dev/zero of=./hello bs=65536 count=491520
 
測試2
- dd if=/dev/zero of=./hello bs=8192 count=3932160 oflag=sync
- dd if=/dev/zero of=./hello bs=65536 count=491520 oflag=sync
 
測試3
- dd if=/dev/zero of=./hello bs=65536 count=491520 oflag=dsync
 
測試4
- dd if=/dev/zero of=./hello bs=65536 count=491520 oflag=direct
 


 
測試結果1 (8KB)
- 32212254720 bytes (32 GB) copied, 260.379 s, 124 MB/s
- cached 55 -> 15007
測試結果1 (64KB)
- 32212254720 bytes (32 GB) copied, 183.879 s, 175 MB/s
- cached 54 -> 15019 (每個server都用了超過10000)
測試結果2 (8KB)
- 382648320 bytes (383 MB) copied, 1771.25 s, 216 kB/s (太久了懶得等..)
- cached 55 -> 69
測試結果2 (64KB)
- 2315583488 bytes (2.3 GB) copied, 1462.57 s, 1.6 MB/s (太久了懶得等..)
測試結果3 (64KB)
- 1516240896 bytes (1.5 GB) copied, 964.448 s, 1.6 MB/s (太久了懶得等..)
測試結果4 (8KB)
- 32212254720 bytes (32 GB) copied, 455.297 s, 70.7 MB/s
測試結果4 (64KB)
- 32212254720 bytes (32 GB) copied, 442.526 s, 72.8 MB/s

 
 

附錄二 - 新增掛載volume

停止服務 (center first)
- service cakeweb stop
- service caked stop
 
新建volume
- 新增brick資料夾 (for all)
- mkdir /mnt/gfbrick/brick4/
- gluster volume create str-vol stripe 3 transport tcp c1:/mnt/gfbrick/brick4/ n1:/mnt/gfbrick/brick4/ n2:/mnt/gfbrick/brick4/
- gluster volume start str-vol
 
備份資料到新volume
- mkdir /mnt/storage2
- mount -t glusterfs c1:/str-vol /mnt/storage2/
- rsync -ravP /mnt/storage/* /mnt/storage2/
 
umout previous Gluster volume
- umount /mnt/storage2/ (only for center)
- umount /mnt/storage/ (for all)
 
mount new Gluster volume (for all)
- mount -t glusterfs c1:/str-vol /mnt/storage/
 
重啟服務
- service caked restart
- service cakeweb restart (center only)
 

附錄三 - 掛載回original CAKE nfs

umount CAKE storage
- umount /mnt/storage

關閉Gluster volume
- gluster volume stop xxx-vol

umount Gluster bricks
- umount /mnt/gfbrick

center original mount
- mount /dev/vg_livecd/lv_mnt_storage /mnt/storage

node original nfs mount
- 相關nfs參數可從 storage.ini 內取得
- mount -t nfs -o rw,vers=3,fg,proto=tcp,noacl,noatime,hard,intr,timeo=600 192.168.10.10:/mnt/storage /mnt/storage

 

附錄四 - testing method of other paper

for writing measurement
- dd if=/dev/zero of=zerofile bs=4M count=2560 conv=fdatasync

for reading measurement
- dd if=zerofile of=/dev/null bs=4M count=2560
 

附錄五 - 移除volume

- - 以dht-vol為例 - -
gluster volume stop dht-vol
gluster volume remove-brick dht-vol c1:/mnt/gfbrick/brick2 force
gluster volume remove-brick dht-vol n1:/mnt/gfbrick/brick2 force
gluster volume remove-brick dht-vol n2:/mnt/gfbrick/brick2 force
gluster volume delete dht-vol
 
 
 
 
 
 
 
 
 
 
 
 
 


 
除非特別註明,本頁內容採用以下授權方式: Creative Commons Attribution-ShareAlike 3.0 License