丟失的ASM磁盤組

1、近日遇到一個(gè)問題， ASM的磁盤組無法掛載，之前是正常的，由于一些其他的操作，數(shù)據(jù)庫啟動(dòng)失敗，當(dāng)問題排除時(shí)候，發(fā)現(xiàn)在數(shù)據(jù)庫整體啟動(dòng)時(shí)，掛載磁盤組的環(huán)節(jié)出現(xiàn)問題。 環(huán)境介紹If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If IfII II II II II II II II II II II II II II II II II II II II II II II II II II II II II II II

2、II II II II II II II II II II硬件：Vmware ESX虛擬機(jī)OS: Red hat li nux 5Oracle versio n: ASM disk是通過 asmlib掛載的這個(gè)磁盤組只有一個(gè)虛擬出的硬盤，是/dev/sdb1.If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If If IfII II II II II II II II II II II II II I

3、I II II II II II II II II II II II II II II II II II II II II II II II II II II II下面是我整個(gè)分析的過程1. 首先通過ASM alert.log，發(fā)現(xiàn)如下錯(cuò)誤，磁盤掛載失敗，無法找到磁盤組1. SQL alter diskgroup DATA mount2. NOTE: cache registered group DATAnumber =1 incarn =0xc28a1e2d= 0xc28a=1 inca3. NOTE: cache began mount (first) of group DATAnumbe

4、r =1 incarn1e2d4. Tue Dec 11 18:06:55 20125. ERROR: no PST quorum in group: required 2, found 06. NOTE: cache dismounting (clean) group 1/0xC28A1E2D (DATA)7. NOTE: dbwr not being msgd to dismount8. NOTE: lgwr not being msgd to dismount9. NOTE: cache dismounted group 1/0xC28A1E2D (DATA)10. NOTE: cach

5、e ending mount (fail) of group DATAnumberrn =0xc28a1e2d11. NOTE: cache deleting context for group DATA 1/0xc28a1e2d12. GMON dismounting group 1 at 8 for pid 17, osid 3216313. ERROR: diskgroup DATA was not mounted14. ORA-15032: not all alterations performed15. ORA-15017: diskgroup DATA cannot be moun

6、ted16. ORA-15063: ASM discovered an insufficient number of disks for diskgroup DATA17. ERROR: alter diskgroup DATA mount2. 首先檢查ASM pfile 文件，未發(fā)現(xiàn)異常1. asm_diskgroups=DATA2. instance_type =asm3. Iarge_pool_size =12M= EXCLUSIVE4. remote_login_passwordfile3. 嘗試通過以下命令檢查磁盤是否物理存在，是如何對(duì)應(yīng)物理設(shè)備的，發(fā)現(xiàn)查詢不到ASM磁盤1. gri

7、dlgto_test $ kfoddisks =all2.2. Non output. gridlgto_test peer$ cd /dev/oracleasm/disks/4. gridlgto_test disks$ ls9.5. Non output11.12.13. gridlgto_test disks$ /etc/init.d/oracleasm listdisks14.14. Non output4. 但是直接檢查物理設(shè)備，/dev/sdb1是存在的，說明OS已經(jīng)識(shí)別該硬盤設(shè)備，只是ASMlib無法正常識(shí)別：1. 查詢對(duì)應(yīng)的物理硬盤2. oracleOEL $ /

8、etc/init.d/oracleasm querydisk -d disk1 Disk DISK1 is a valid ASM disk ondevice 8,173. oracleOEL $ ls -l /dev/ |grep 8|grep 17 brw-r1 rootdisk 8,17 Oct 16 14:01sdb14.4. rootlgto_test # ls -lst /dev/sd*5. 0brw-r1root disk8,0 Dec 11 19:29 /dev/sda6. 0brw-r1root disk8,2 Dec 11 19:29 /dev/sda27. 0brw-r1

9、root disk8,16 Dec 11 19:29 /dev/sdb8. 0brw-r1root disk8,17 Dec 11 19:29 /dev/sdb1Thisis the missed diskgroup9. 0 brw-r1 root disk 8, 1 Dec 11 11:29 /dev/sda15. 起先是考慮是否是磁盤頭損害，導(dǎo)致無法ASMlib識(shí)別該磁盤，dump磁盤頭發(fā)現(xiàn)沒有問題.1. #od -c /dev/sdbl2. 3. 0000040 O R C L D I S K14. 5. 7760040 O R C L D I S K1這里補(bǔ)充下，如果磁盤頭信息丟失，將

10、會(huì)顯示如下1. 0000040 O R C L D I S K 0 0 0 0 0 0 0 0如果顯示這個(gè)結(jié)果，需要通過以下方式 renamedisk，具體可以參考文檔Oracleasm ListdisksCannot See Disks (Doc ID 392527.1)1. Use the oracleasm renamedisk utility to add an asmlib label to thedisk:2. /etc/init.d/oracleasm renamedisk /dev/ 3. If it fails, use the -f switch:4. /etc/init.

11、d/oracleasm renamedisk -f /dev/ 6. 重啟ASMLib，檢查是否是 ASMLib問題1. rootlgto_test # /etc/init.d/oracleasm restart2. Dropping Oracle ASMLib disks:3. OK 4. Shutting down the Oracle ASMLib driver: FAILED檢查文件系統(tǒng)oracleasm文件系統(tǒng)已經(jīng)成功掛載1. rootlgto_test # df -ha2. Filesystem Size Used Avail Use% Mounted on3. 4. oracle

12、asmfs000 - /dev/oracleasm7. 檢查/dev/sdb1 狀態(tài)，查看是否已經(jīng) marked為ASM disk，顯示已經(jīng)標(biāo)記成功1. rootlgto_test # oracleasm querydisk /dev/sdb12. Device /dev/sdb1 is marked an ASM disk with the label DATADG013.3. rootlgto_test # /sbin/service oracleasm scandisks4. Scanning the system for Oracle ASMLib disks:5. OK 8. roo

13、tlgto_test # /etc/init.d/oracleasm listdisks9. Non output-8. 檢查rpm package也沒有問題1. gridlgto_test $ rpm -qa|grep oracleasm2. oracleasmlib-2.0.4-1.el53. oracleasm-support-2.1.7-1.el54. oracleasm-2.6.18-308.el5-2.0.5-1.el59. 收集Kfed logs,沒有檢查到異常新信息。1. rootlgto_test # /oracle/ora11g/product/app/grid/bin/k

14、fed read /dev/sdb12.3. kfbh.endian:1 ; 0x000: 0x014.5. kfbh.hard:6.6. kfbh.type:8.7. kfbh.datfmt:10.8. kfbh.block.blk: x012.9. kfbh.block.obj:UMB=0x014.130 ; 0x001: 0x821 ; 0x002: KFBTYP_DISKHEAD1 ;0x003: 0x010 ; 0x004:2147483648 ; 0x008:T=0 NUMB=0TYPE=0x8 N15.kfbh.check:3351358462 ; 0x00c: 0xc7c1ab

15、fe16.17.kfbh.fcn.base:0 ; 0x010: 0x0000000018.19. kfbh.fcn.wrap:20.20. kfbh.spare1:22.23.kfbh.spare2:0;0x014: 0x000000000;0x018: 0x000000000 ; 0x01c: 0vstr: ORCLDISKDATADG01 ; 0x000:kfdhdb.driver.reservedO: 1096040772 ; 0x008: 0x41544144kfdhdb.driver.reserved1: 825247556 ;

16、0x00c: 0x31304744kfdhdb.driver.reserved2:kfdhdb.driver.reserved3:kfdhdb.driver.reserved4:kfdhdb.driver.reserved5:0;0x010: 0x000000000 ; 0x014: 0x000000000 ; 0x018: 0x000000000 ; 0x01c: 0x00000000pat:186646528 ; 0x020: 0x0b200000kfdhdb.dsknum:0 ; 0x024: 0x0000ERNALkfdhdb.grptyp:1 ; 0x026: KFDGTP_EXTB

17、ERkfdhdb.hdrsts:3 ; 0x027: KFDHDR_MEMkfdhdb.dskname:DATADG01 ; 0x028:kfdhdb.grpname:DATA ; 0x048:kfdhdb.fgname:DATADG01 ; 0x068:kfdhdb.capname:;0x088:kfdhdb.crestmp.hi:32977140 ; 0x0a8:length =16length =8length =4length =8length =05.46.

18、6.57.kfdhdb.crestmp .Io:1642529792 ; 0x0ac:USEC=OxO M8.89.SEC=0x1c1 SECS=0x1e MINS = 0x18HOUR=0x14kfdhdb.mntstmp.hi:32977140 ; OxObO:DAYS= 0x7 MNTH=0xc YEAR=0x7dcUSEC=0x0 Mkfdhdb.mn

19、tstmp.lo:1664549888 ; 0x0b4:MINS = 0x18SEC=0x1c1 SECS=0x33kfdhdb.secsize:kfdhdb.blksize:kfdhdb.ausize: kfdhdb.mfact: kfdhdb.dsksize: kfdhdb.pmcnt: kfdhdb.fstlocn: kfdhdb.altlocn: kfdhdb.f1b1locn: kfdhdb.redomirrorsO:kfdhdb.redomirrors1:kfdhdb.redomirrors2:kfdhdb.redomirrors3:kfdhdb.dbcompat:512 ; 0x

20、0b8: 0x02004096 ; OxOba: 0x10001048576 ; OxObc: 0x00100000113792 ; OxOcO: 0x0001bc80204797 ; 0x0c4: 0x00031ffd3 ; 0x0c8: 0x000000031 ; OxOcc: 0x000000012;OxOdO: 0x000000022 ; 0x0d4: 0x000000020 ; 0x0d8: 0x00000 ; OxOda: 0x00000 ; OxOdc: 0x00000 ; OxOde: 0x0000168820736 ; OxOeO: 0x0a1000003

21、.000018.HOUR=0x14kfdhdb.grpstmp.hi:32977140 ; 0x0e4:DAYS=0x7 MNTH=0xc YEAR=0x7dcUSEC=0x0 Mkfdhdb.grpstmp.lo:1642390528 ; 0x0e8:SEC=0x139 SECS=0x1e MINS = 0x18kfdhdb.vfstart:0 ; 0x0ec: 0x00000000kfdhdb.vfend:0 ; 0x

22、0f0: 0x00000000kfdhdb.spfile:58 ; 0x0f4: 0x0000003akfdhdb.spfflg:1 ; 0x0f8: 0x00000001kfdhdb.ub4spare0:0 ; 0x0fc: 0x00000000kfdhdb.ub4spare1:0;0x100: 0x00000000kfdhdb.ub4spare2:0;0x104: 0x00000000kfdhdb.ub4spare3:0;0x108: 0x00000000kfdhdb.ub4spare4:0 ; 0x10c: 0x00000000kfdhdb.ub4spare5:0;0x110: 0x00

23、000000kfdhdb.ub4spare6:0;0x114: 0x00000000119.kfdhdb.ub4spare8:0 ; 0x11c: 0x00000000120.121.kfdhdb.ub4spare9:0;0x120: 0x00000000122.123.kfdhdb.ub4spare10:0;0x124: 0x00000000124.125.kfdhdb.ub4spare11:0;0x128: 0x00000000126.127.kfdhdb.ub4spare12:0 ; 0x12c: 0x00000000128.129.kfdhdb.ub4spare13:0;0x130:

24、0x00000000130.131.kfdhdb.ub4spare14:0 ; 0x134: 0x00000000132.133.kfdhdb.ub4spare15:0;0x138: 0x00000000134.135.kfdhdb.ub4spare16:0 ; 0x13c: 0x00000000136.137.kfdhdb.ub4spare17:0;0x140: 0x00000000138.139.kfdhdb.ub4spare18:0 ; 0x144: 0x00000000140.141.kfdhdb.ub4spare19:0;0x148: 0x00000000142.143.kfdhdb

25、.ub4spare20:0 ; 0x14c: 0x00000000144.145.kfdhdb.ub4spare21:0;0x150: 0x00000000146.147.kfdhdb.ub4spare22:0 ; 0x154: 0x00000000149. kfdhdb.ub4spare23:150.150. kfdhdb.ub4spare24:152.151. kfdhdb.ub4spare25:154.152. kfdhdb.ub4spare26:156.153. kfdhdb.ub4spare27:158.154. kfdhdb.ub4spare28:160.155. kfdhdb.u

26、b4spare29:162.156. kfdhdb.ub4spare30:164.157. kfdhdb.ub4spare31:166.158. kfdhdb.ub4spare32:168.159. kfdhdb.ub4spare33:170.160. kfdhdb.ub4spare34:172.161. kfdhdb.ub4spare35:174.162. kfdhdb.ub4spare36:;0x158: 0x00000000;0x15c: 0x00000000;0x160: 0x00000000;0x164: 0x00000000;0x168: 0x00000000;0x16c: 0x0

27、0000000;0x170: 0x00000000;0x174: 0x00000000;0x178: 0x00000000;0x17c: 0x00000000;0x180: 0x00000000;0x184: 0x00000000;0x188: 0x0000000000000000000000004.205.kfdhdb.ub4spare37:0;0x190: 0x00000000

28、kfdhdb.ub4spare38:0 ; 0x194: 0x00000000kfdhdb.ub4spare39:0;0x198: 0x00000000kfdhdb.ub4spare40:0 ; 0x19c: 0x00000000kfdhdb.ub4spare41:0;0x1a0: 0x00000000kfdhdb.ub4spare42:0;0x1a4: 0x00000000kfdhdb.ub4spare43:0;0x1a8: 0x00000000kfdhdb.ub4spare44:0 ; 0x1ac: 0x00000000kfdhdb.ub4spare45:0;0x1b0: 0x000000

29、00kfdhdb.ub4spare46:0;0x1b4: 0x00000000kfdhdb.ub4spare47:0;0x1b8: 0x00000000kfdhdb.ub4spare48:0 ; 0x1bc: 0x00000000kfdhdb.ub4spare49:0;0x1c0: 0x00000000kfdhdb.ub4spare50:0;0x1c4: 0x00000000kfdhdb.ub4spare51:0 ; 0x1c8: 0x00000000207.kfdhdb.ub4spare52:0 ; 0x1cc: 0x00000000208.209.kfdhdb.ub4spare53:0;0

30、x1d0: 0x00000000210.211.kfdhdb.acdb.aba.seq:0;0x1d4: 0x00000000212.213.kfdhdb.acdb.aba.blk:0;0x1d8: 0x00000000214.215.kfdhdb.acdb.ents:0 ; 0x1dc: 0x0000216.217.kfdhdb.acdb.ub2spare:0 ; 0x1de: 0x000010. 階段總結(jié)，通過以上的分析，得出以下總結(jié)1. ASMLib 正常2. RPM包正常3. 磁盤頭沒有損壞和丟失信息4. 該硬件已經(jīng)被系統(tǒng)正常識(shí)別當(dāng)前問題就是為什么 ASMLib不能正常掃描并識(shí)別到該硬盤11. 最后在檢查文件/etc/sysconfig/