玉柴私有云實(shí)施方案參考h3c2014onestor運(yùn)維v3

1、H3CONEStor運(yùn)維大H3CTechnologies.1概2Arp-.與ss命令對(duì)H3CONEStor運(yùn)維大H3CTechnologies.1概2Arp-.與ss命令對(duì)控制功能3.磁盤(pán)擴(kuò)集群擴(kuò)網(wǎng)絡(luò)變handyip變clusterip變 4日志介日志類(lèi)日志收告警查查看告郵件告告警查查看告郵件告5安裝InfiniBand 驅(qū)6獲取及安使用方參數(shù)介使用限應(yīng)用舉使用方應(yīng)用舉使用方安裝選擇硬掃描分區(qū)寫(xiě)入數(shù)7附12Arp-所有 12Arp-所有 erface=eth0 erface:eth0,datalinktype:EN10MB HYPERLINK http:/w/ 124694567826pack

2、etsreceivedbyfilter,0packetsdroppedbyEndingarp-scan1.8.1:256hostsscannedin26packetsreceivedbyfilter,0packetsdroppedbyEndingarp-scan1.8.1:256hostsscannedin1.256seconds(203.82hosts/sec).26查看測(cè)試系統(tǒng)內(nèi)某個(gè)IP rootubuntu3:#sudoarp-scan-Ieth0erface:eth0,datalinktype:EN10MB HYPERLINK http:/w/ 1packetsreceivedbyfi

3、lter,0packetsdroppedbyEndingarp-scan1.8.1:1hostsscannedin0.415seconds(2.41hosts/sec).0Nets ss Linux 系統(tǒng)都會(huì)默認(rèn)包含 命令，但并非所有系統(tǒng)都會(huì)默認(rèn)包含 ss 命令。是 net-tools 工具集中的一員，而 ss 命令是 iproute 工具集中的一員，如果你無(wú)法使用 ss 命令那么可能是缺少了 iproute，需要安裝一下：ss命令可以用來(lái)獲取socket統(tǒng)計(jì)信息，它可以顯示和nets 類(lèi)似的內(nèi)容。但ss的優(yōu)勢(shì)在于它能夠顯更詳細(xì)的有關(guān)TCP和連接狀態(tài)的信息，而且比nets 更快速更高效ss快的

4、秘訣在于，它利用到了TCP協(xié)議ss快的秘訣在于，它利用到了TCP協(xié)議棧中tcp_diag。tcp_diag是一個(gè)用于分析統(tǒng)計(jì)的模塊， 可以獲得Linux 內(nèi)核中第一手的信息，這就確保了ss的快捷高效。當(dāng)然，如果你的系統(tǒng)中沒(méi)有 tcp_diag，ss也可以正常運(yùn)行，只是效率會(huì)變得稍慢。當(dāng)前服務(wù)器的網(wǎng)絡(luò)連接統(tǒng)計(jì)#ss 如果只想查看 TCP sockets，那么使用-ta 選項(xiàng)；如果只想查看 UDP sockets，那么使用-ua 選項(xiàng)；如果只想查看 RAW sockets，那么使用-wa 選項(xiàng)；如果只想查看 UNIX sockets，那么使用-xa 選項(xiàng)。命令對(duì)照表控制功 控制功 格式：iptab

5、les-tDchainCRETIRIA-j-ttable ：3個(gè)filternat ptablesAINPUTspTCPjDROPptables-AINPUT-s/24-pTCPptablesAINPUTspTCPjDROPptables-AINPUT-s/24-pTCP-j # ptablesAINPUTme -ptablesAINPUTptcpieth0dport4848jACCEPT#開(kāi)發(fā)TCPU的4848pre-upiptables-restore設(shè)這個(gè)協(xié)議，進(jìn)來(lái)的為 8（），出去的為 0(響應(yīng)).要 8 出去,允許 0 進(jìn)來(lái)。在出去的端口上：iptablesAOUTPUTpicmpi

6、cmp-type8j在進(jìn)來(lái)的端口上：iptablesAINPUTpicmpicmp-type0j33息physicaldrive2I:2:2(port2I:box2:bay2,SATA,1TB,息physicaldrive2I:2:2(port2I:box2:bay2,SATA,1TB,rootceph4:#hpssaclictrl slot=3createtype=ldWarning: Creation of this logical drive has caused array letters e arrayG(SATA,UnusedSpace:0 logicaldrive7(931.5G

7、B,RAID0,physicaldrive2I:2:2(port2I:box2:bay2,SATA,1TB,（9） 此時(shí)繼續(xù)執(zhí)行4.2.2（9） 此時(shí)繼續(xù)執(zhí)行.1 3.2.1 3.2.2 3.2.3 RBD （1） 無(wú)用戶使用3.2.2 3.2.3 RBD （1） 無(wú)用戶使用（2） 若有用戶在使用（2） 若有用戶在使用 3.3.1 handy ip 而 a.初始handy節(jié)點(diǎn)availableonyour#Thisfiledescribesthe#andhowtoactivatethem.Formoreinformation,#Theackautoifaceloinet#Theprimarya

8、utoifaceeth0inetbroadcast gateway #Theackautoifaceloinet#Theprimaryautoifaceeth0inetbroadcast gateway #dns-*optionsareimplemented dns-nameservers autobytheresolvconfpackage,ififaceeth1inetaddressauto eth2ifaceeth2inetaddressnetmask availableonyour#Thisfiledescribestheandhowtoactivatethem.Formoreinfo

9、rmation,Theackautoifaceloinet#Theprimaryautoifaceeth0inetbroadcast gateway #ifaceeth0inetbroadcast gateway #dns-*optionsareimplementedbytheresolvconfpackage,ifinstalled dns-nameservers autoifaceeth1inetaddressauto eth2ifaceeth2inetaddressnetmask #Thisfiledescribestheavailableonyour#andhow#Thisfilede

10、scribestheavailableonyour#andhowtoactivatethem.Formoreinformation,Theack#autoifaceloinet#Theprimaryautoifaceeth0inetbroadcast gateway #dns-*optionsareimplemented dns-nameservers autobytheresolvconfpackage,ififaceeth1inetaddressauto eth2ifaceeth2inetaddressnetmask availableonyour#Thisfiledescribesthe

11、andhowtoactivatethem.Formoreinformation,availableonyour#Thisfiledescribestheandhowtoactivatethem.Formoreinformation,Theackautoifaceloinet#Theprimaryautoifaceeth0inetbroadcast gateway #dns-*optionsareimplemented dns-nameservers autobytheresolvconfpackage,ififaceeth1inetaddress auto eth2ifaceeth2ineta

12、ddressnetmask handy handyiphandyip3.3.2 cluster ip rootceph1:/home/ceph-cluster# vi #Thisfiledescribestheavailableonyourandhowtoactivatethem.Formoreinformation,Theack rootceph1:/home/ceph-cluster# vi #Thisfiledescribestheavailableonyourandhowtoactivatethem.Formoreinformation,Theackautoifaceloinet#Th

13、eprimaryautoifaceeth0inetbroadcast gateway #dns-*optionsareimplemented dns-nameservers bytheresolvconfpackage,ifautoifaceeth1inetaddress auto eth2ifaceeth2inetaddress netmask 0(0)56(84)bytesof-sistics-2packetstransmitted,0received,100%packetloss,time通60 (60) 56(84) bytes of time=0.386 time=0.257 -si

14、stics-3packetstransmitted,3received,0%packetloss,time 60 (60) 56(84) bytes of time=0.386 time=0.257 -sistics-3packetstransmitted,3received,0%packetloss,time #Thisfiledescribestheavailableonyourandhowtoactivatethem.Formoreinformation,Theackautoifaceloinet#Theprimaryautoifacevswitch0inetbroadcast gate

15、way #dns-*optionsareimplemented dns-nameservers autobytheresolvconfpackage,ifaceeth0inetaddressnetmask autoifaceeth1inetaddress netmask autoifaceeth2inetaddress auto eth3ifaceeth3inets address netmask address auto eth3ifaceeth3inets address netmask fsid = 1575eefa-d8c4-48e9-8396-1038beb2dba5 mon_ini

16、tial_members = cvknode1, cvknode2, cvknode3 mon_host = 5,6,7auth_cluster_required = auth_service_required = nt_required=filestore_xattr_use_omap=osd_journal_size = 10240 public_network = /24 mon_config_key_max_entry_size = filestore_merge_threshold = filestore_split_multiple = filestore_fd_cache_siz

17、e = filestore_fd_cache_shards = osd_enable_op_tracker = rbd_cache=host=keyring=log_file=rgw_frontends=civetwebrgw_dns_name=osd_crush_update_on_start = cluster_network/24 修改為cluster_networkosd_crush_update_on_start = cluster_network/24 修改為cluster_network fsid = 7f3d1904-ee11-47e6-968e-ff265e47510a mo

18、n_initial_members = ceph1, ceph2, ceph3mon_host = auth_cluster_required = cephx auth_service_required = nt_required=filestore_xattr_use_omap = osd_journal_size=public_network=cluster_network=mon_config_key_max_entry_size = filestore_merge_threshold = filestore_split_multiple = filestore_fd_cache_siz

19、e = filestore_fd_cache_shards = osd_enable_op_tracker = rbdcache= = 5(5)56(84)bytesof-istics-2packetstransmitted,0received,100%packetloss,time5(5)56(84)bytesof-istics-2packetstransmitted,0received,100%packetloss,time通35 (35) 56(84) bytes of time=0.119 更換故障的mon 3.4.1 ONEStorUbuntu14.04.3: 1、 在backupN

20、ode1上增加mon進(jìn)程2、 刪除掉故障節(jié)點(diǎn)primaryNode1的mon進(jìn)程 / cp/etc/ceph/ceph.conf/home/ceph-rootprimaryNode1:/etc/ceph#ceph-deploy-overwrite-confconfigprimaryNode2primaryNode3 cat ceph-deploy-overwrite-confmon cat ceph-deploy-overwrite-confmonaddceph -sStep6：刪除掉primaryNode1ceph-deploymondestroyStep7:在所有節(jié)點(diǎn)更新moncp/etc

21、/ceph/ceph.conf/home/ceph-rootprimaryNode1:/etc/ceph#ceph-deploy-overwrite-confconfigprimaryNode2primaryNode3Step8：更新clusterconfig，保證HandyrootprimaryNode1:/etc/ceph#cephconfig-keyget/24,ntp_close:1,mon_ip:83,82,mon_fqdns:backupNode1,primaryNode2,primaryNode3, public_network: /24 3.4.2 ONEStor 部署在CAS

22、 16節(jié)點(diǎn)的ONEStor集群cvknode1、cvknode2、cvknode3cvknode15、1、 在cvknode4上增加mon2、除掉故1、 在cvknode4上增加mon2、除掉故障節(jié)點(diǎn)cvknode1的mon進(jìn)程。Step1:在Handy上節(jié)點(diǎn)，先將cvknode4 Step2:更新HandyBceph-deploy-overwrite-confconfigpushcvknode2cvknode3cvknode4cvknode5 cvknode6cvknode7cvknode8cvknode9cvknode10cvknode11cvknode12cvknode14cvknode

23、15 Step3:cat Step4:在cvknode4上增加mon節(jié)點(diǎn)rootcvknode1:/etc/ceph#ceph-deploy-overwrite-confmon add cephStep6:刪除掉cvknode1上的monrootcvknode1:/etc/ceph#ceph-deploymondestroyStep7:在所有節(jié)點(diǎn)更新mon文件rootcvknode1:/etc/ceph#cpceph.conf/home/ceph-C、 rootcvknode1:/etc/ceph#cephconfig-keygetrootcvknode1:/etc/ceph#cephconf

24、ig-keycluster_network:/24,mon_ip:6,7,mon_fqdns:cvknode4,cvknode2, cvknode3, public_network: /24Step9:rootcvknode1:/etc/ceph#cephconfig-keygetrootcvknode1:/etc/ceph#cephconfig-keycluster_network:/24,mon_ip:6,7,mon_fqdns:cvknode4,cvknode2, cvknode3, public_network: /24Step9:Handy界面是否能夠正常顯示新增的MON內(nèi)內(nèi)日志4.

25、1.2 4.2.1 4.2.2 從handy在在在20GB當(dāng)當(dāng)提示）提示）4.4.1 NTP4.4.1 NTP restrictIPmasknetmask_IP restrictIPmasknetmask_IP noquery：不提供C ntnotrust：該C nt除非通過(guò)認(rèn)證，否則該C nt2server NTPserver在 客戶端與NTPServer1ntpdate ntpdateNTPServer時(shí)間一致。如下圖所示，offset 是指與NTP Server 的時(shí)間差。 ntpdate不會(huì)考慮其他程序是否會(huì)陣痛，直接調(diào)整時(shí)間。一個(gè)是校準(zhǔn)時(shí)間，一個(gè)是ONEStor集群開(kāi)啟NTP 服務(wù)后

26、，4.4.3 NTP 4.4.3 NTP 4.4.4 ONEStoreNTP客戶端向NTPServer進(jìn)行時(shí)間同步的過(guò)程，這 節(jié)點(diǎn)為NTP +MON2節(jié)點(diǎn)MON3節(jié)點(diǎn)MON1節(jié)點(diǎn) 節(jié)點(diǎn)只做NTP 4.4.5 ONEStor與CAS先部署 CAS 集群，設(shè)置NTPServerIP地址，再部署ONEStorONEStorCAS集群，設(shè)置NTPServerIP（2）ONEStorCAS集群，配置NTPServerIP 4.5.1 1. onestor-clidestroyhosts 集群ip(用逗號(hào)隔開(kāi)onestor-clidestroy-hosts2. （1） onestor-cliprobeha

27、ndy節(jié)點(diǎn)（2） onestor-clionestor-cliprobeonestor-cli2. （1） onestor-cliprobehandy節(jié)點(diǎn)（2） onestor-clionestor-cliprobeonestor-cli 55.1.1檢查InfiniBand ： 件InfiniBand5.1.3安裝InfiniBand ： 件InfiniBand5.1.3安裝InfiniBand ./5.1.4InfiniBand（1） 開(kāi)啟openibd （2） 啟動(dòng)opensm理 （3） 查看網(wǎng)口狀態(tài) （2） 啟動(dòng)opensm理 （3） 查看網(wǎng)口狀態(tài) （4） 配置vi（5） 測(cè)試網(wǎng)口速率

28、服務(wù)器端ifdownib0&ifupib0 ifconfig ib05.1.5m4tk8.4dpatchflex swiggfortrandkmslibgfortran3utomakepkg-5.1.5m4tk8.4dpatchflex swiggfortrandkmslibgfortran3utomakepkg-libxml2tclgraphviztcl8.4tkapt-getinstallym4tk8.4dpatchflex LinuxIOLinuxIO-d-k1為-d -m 為 -d-x-k1 -c110用增加。await的大小一般取決于服務(wù)時(shí)間(svctm)I/OI/OI/O HP6.

29、1.1hpssacliI/O HP6.1.1hpssacli6.1.2 6.1.3# hpssaclictrlallshowSmartArrayP440arinSlot0ControllerS us:CacheS us: Battery/CapacitorS # hpssaclictrlallshowSmartArrayP440arinSlot0ControllerS us:CacheS us: Battery/CapacitorS us:SmartArrayP840inSlotControllerS us:CacheS us: Battery/CapacitorS us:#hpssaclic

30、trlslot=0showconfig#hpssaclictrlslot=0showconfigdetail|grepDiskName:DiskName:DiskName:DiskName:DiskName:DiskName:DiskName:DiskName:#hpssaclictrlslot=0ldallSmartArrayP440arinSlot0arraylogicaldrive1(1.1TB,RAID0,arraylogicaldrive2(1.1TB,RAID0,arraylogicaldrive3(1.1TB,RAID0,arraylogicaldrive4(1.1TB,RAID

31、0,arraylogicaldrive5(1.1logicaldrive3(1.1TB,RAID0,arraylogicaldrive4(1.1TB,RAID0,arraylogicaldrive5(1.1TB,RAID0,arraylogicaldrive6(1.1TB,RAID0,arraylogicaldrive7(1.1TB,RAID0,arraylogicaldrive8(1.1TB,RAID0,查看某個(gè)#hpssaclictrlslot=0arrayAldall/查看slot0 陣列A SmartArrayP440arinSlot0arraylogicaldrive1(1.1TB,

32、RAID0,hpssaclictrlslot=0arrayApdallshow/查看slot0 陣列A SmartArrayP440arinSlot0arrayphysicaldrive1I:3:1(port1I:box3:bay1,SAS,1200.2GB,#hpssaclictrlslot=0logicaldrive6#hpssaclictrlslot=0pdallshowphysicaldrive1I:3:1(port1I:box3:bay1,1200.2GB):physicaldrive1I:3:2(port1I:box3:bay2,1200.2GB):physicaldrive1I:

33、3:3(port1I:box3:bay3,1200.2GB):physicaldrive1I:3:4(port1I:box3:bay4,1200.2GB):physicaldrive2I:3:6(port2I:box3:bay6,1200.2GB):physicaldrive2I:3:5(port2I:box3:bay5,1200.2GB):physicaldrive2I:3:8(port2I:box3:bay8,1200.2GB):physicaldrive2I:3:7(port2I:box3:bay7,1200.2GB):#hpssaclictrlslot=0pdall#hpssaclictrlallshowconfigdetail|grep-i#hpssaclictrlslot=0pdall#hpssaclictrlallshowconfi