CUDB和PG閏秒調(diào)整方案(2015.6.29)

1、 中國(guó)移動(dòng)通信集團(tuán)廣東有限公司網(wǎng)絡(luò)管理中心核網(wǎng)絡(luò)維護(hù)室CUDB和PG閏秒調(diào)整方案中國(guó)移動(dòng)廣東公司網(wǎng)管維護(hù)中心核心網(wǎng)絡(luò)維護(hù)室2015年6月 目錄一、網(wǎng)絡(luò)變更簡(jiǎn)報(bào)41.1網(wǎng)絡(luò)變更背景41.2網(wǎng)絡(luò)變更內(nèi)容4二、變更風(fēng)險(xiǎn)點(diǎn)與業(yè)務(wù)影響評(píng)估52.1本次網(wǎng)絡(luò)變更的主要風(fēng)險(xiǎn)點(diǎn)52.2本次變更實(shí)施人員52.3是否為首次變更52.4前期測(cè)試結(jié)果52.5業(yè)務(wù)影響評(píng)估52.5.1廠家意見(jiàn)52.5.2變更負(fù)責(zé)人意見(jiàn)52.6 綜合影響評(píng)估62.6.1 計(jì)費(fèi)影響62.6.2 綜資及話務(wù)網(wǎng)管影響62.6.3 支撐系統(tǒng)調(diào)整62.6.4 性能統(tǒng)計(jì)影響62.6.5 業(yè)務(wù)流程影響62.7 領(lǐng)導(dǎo)決策點(diǎn)6三、操作內(nèi)容與流程63.1 相關(guān)

2、拓?fù)鋱D63.1 操作內(nèi)容與流程7四、詳細(xì)實(shí)施步驟84.1 變更前準(zhǔn)備84.2 詳細(xì)實(shí)施步驟9五、測(cè)試與驗(yàn)證方案105.1 驗(yàn)證方案（定期撥測(cè)）105.1 驗(yàn)證方案（定期指標(biāo)）10六、應(yīng)急方案116.1 本次變更可能出現(xiàn)的問(wèn)題116.2 應(yīng)急與倒回方案11七、BIBYSITTING監(jiān)測(cè)要求13八、聯(lián)系人表13九、備注14一、網(wǎng)絡(luò)變更簡(jiǎn)報(bào)1.1網(wǎng)絡(luò)變更背景本次操作是針對(duì)新增NGBOSS服務(wù)開(kāi)通主機(jī)的靜態(tài)路由配置的方案1.2網(wǎng)絡(luò)變更內(nèi)容修改網(wǎng)絡(luò)配置 添加和NGBOSS相關(guān)的兩條靜態(tài)路由變更涉及網(wǎng)元網(wǎng)絡(luò)變更內(nèi)容網(wǎng)絡(luò)影響情況變更操作時(shí)間業(yè)務(wù)影響時(shí)間業(yè)務(wù)受影響網(wǎng)元修改PG和CUDB的NTP配置無(wú)2015/

3、06/30 00:00 05:00無(wú)無(wú)二、變更風(fēng)險(xiǎn)點(diǎn)與業(yè)務(wù)影響評(píng)估2.1本次網(wǎng)絡(luò)變更的主要風(fēng)險(xiǎn)點(diǎn)序號(hào)風(fēng)險(xiǎn)點(diǎn)具體描述變更負(fù)責(zé)人意見(jiàn)1局?jǐn)?shù)據(jù)變更修改NTP服務(wù)器啟動(dòng)配置22.2本次變更實(shí)施人員2.3是否為首次變更是2.4前期測(cè)試結(jié)果修改配置后，觀察設(shè)備的性能統(tǒng)計(jì)和告警是否正常2.5業(yè)務(wù)影響評(píng)估2.5.1廠家意見(jiàn)預(yù)計(jì)不會(huì)對(duì)網(wǎng)絡(luò)業(yè)務(wù)產(chǎn)生不良影響。2.5.2變更負(fù)責(zé)人意見(jiàn)預(yù)計(jì)不會(huì)對(duì)網(wǎng)絡(luò)業(yè)務(wù)產(chǎn)生不良影響。2.6 綜合影響評(píng)估2.6.1 計(jì)費(fèi)影響不會(huì)對(duì)計(jì)費(fèi)產(chǎn)生影響變化。2.6.2 綜資及話務(wù)網(wǎng)管影響相關(guān)網(wǎng)元信息在綜資和話務(wù)網(wǎng)管中不需要更新。2.6.3 支撐系統(tǒng)調(diào)整不需要在支撐系統(tǒng)上補(bǔ)充配置，例如億陽(yáng)主動(dòng)性能

4、監(jiān)控、IPNET網(wǎng)管等。2.6.4 性能統(tǒng)計(jì)影響不會(huì)對(duì)網(wǎng)元或網(wǎng)管上的性能統(tǒng)計(jì)報(bào)表產(chǎn)生影響。2.6.5 業(yè)務(wù)流程影響不會(huì)對(duì)業(yè)務(wù)流程產(chǎn)生變化。2.7 領(lǐng)導(dǎo)決策點(diǎn)序號(hào)需要領(lǐng)導(dǎo)決策點(diǎn)具體描述變更負(fù)責(zé)人意見(jiàn)12三、操作內(nèi)容與流程3.1 相關(guān)拓?fù)鋱D3.1 操作內(nèi)容與流程序號(hào)操作內(nèi)容實(shí)施人實(shí)施時(shí)間結(jié)果確認(rèn)1修改NTP服務(wù)器啟動(dòng)配置2015/06/30 00:00 2四、詳細(xì)實(shí)施步驟4.1 變更前準(zhǔn)備PG:序號(hào)變更準(zhǔn)備項(xiàng)實(shí)施人配合人實(shí)施時(shí)間結(jié)果確認(rèn)1設(shè)備廠家網(wǎng)元類(lèi)型序號(hào)核查項(xiàng)核查及判定標(biāo)準(zhǔn)備注說(shuō)明愛(ài)立信PG1dvemon status n allCAI3G ports should be E/E on all

5、 PL.MML ports should be E/E on all PL.CLI ports should be D/D on all PL.OAM ports should be E/E on all PL.Application server should be E/E on all PL.Deployment error number should be 0 on all PL.ESA processes should be UP on all the nodes.License server should be UP on both SC.Block proxy should be

6、UP on PL3.Reporter should be UP on all PL.Zookeeper should be UP on both SC and PL-3.EMA愛(ài)立信PG2fmacivealarmsThere should be no critical alarms.EMA2檢查PG的proclog，確認(rèn)各類(lèi)指令的發(fā)放成功率指標(biāo)檢查，精確到指令級(jí)別（ADD_SUB、LST_SUB等）3備份所有GEP3板卡的NTP配置cp /etc/init.d/ntp /etc/init.d/ntp.bak4CUDB:序號(hào)變更準(zhǔn)備項(xiàng)實(shí)施人配合人實(shí)施時(shí)間結(jié)果確認(rèn)1設(shè)備廠家網(wǎng)元類(lèi)型序號(hào)核查項(xiàng)核查及

7、判定標(biāo)準(zhǔn)備注說(shuō)明愛(ài)立信CUDB1cudbSystemStatus(1) The time and date is right,(2)CUDB software version is right, (3)11B:System Monitor threads status on each system controller board are OK; 13B:System Monitor leader and BC Server status are OK.(4)Clusters status on each PL and DS is OK, the memory usage lower than

8、 80% (5) There are 4 NDBs process are running on PL Unit and 2 NDBs process are running on each DS Unit, (6) Replication Channels are OK, 'M': Master, 'Sn': Slave replication channel #n is active. (7)There is no new alarm is printed, more information refer to chapter 2.2.16 (5) MySQL

9、 servers connection are listed as “OK”; (6) CUDB Processes in OAMs, PLs and DSs are listed as “running”;愛(ài)立信CUDB2fmacivealarmsThere should be no critical alarms.23備份所有GEP3板卡的NTP配置cp /etc/sysconfig/ntp /etc/sysconfig/ntp.bak44.2 詳細(xì)實(shí)施步驟步驟要求細(xì)化到指令級(jí)。PG:序號(hào)詳細(xì)實(shí)施步驟（細(xì)化到指令級(jí)）實(shí)施人配合人實(shí)施時(shí)間結(jié)果確認(rèn)1登錄PG的第一塊GEP32更改NTP工作模式

10、，修改ntp配置文件sed -i 's/-ng/-xng/g' /etc/init.d/ntp3然后重啟板卡上的ntp服務(wù)service ntp restart4在其余板卡重復(fù)上述步驟CUDB:序號(hào)詳細(xì)實(shí)施步驟（細(xì)化到指令級(jí)）實(shí)施人配合人實(shí)施時(shí)間結(jié)果確認(rèn)1登錄CUDB的備用SC2更改NTP工作模式，修改ntp配置文件/etc/sysconfig/ntp，將“NTPD_OPTIONS="-g -u ntp:ntp"”改為“NTPD_OPTIONS="-x -g -u ntp:ntp"sed -i 's/-g -u ntp:ntp /

11、-x -g -u ntp:ntp/g' /etc/sysconfig/ntp3然后重啟板卡上的ntp服務(wù)rcntp restart4等2分鐘5登錄CUDB的主用SC6更改NTP工作模式，修改ntp配置文件/etc/sysconfig/ntp，將“NTPD_OPTIONS="-g -u ntp:ntp"”改為“NTPD_OPTIONS="-x -g -u ntp:ntp"sed -i 's/-g -u ntp:ntp /-x -g -u ntp:ntp/g' /etc/sysconfig/ntp7然后重啟板卡上的ntp服務(wù)rcntp

12、 restart五、測(cè)試與驗(yàn)證方案5.1 驗(yàn)證方案（定期撥測(cè)）序號(hào)撥測(cè)內(nèi)容網(wǎng)元驗(yàn)證頻度驗(yàn)證結(jié)果驗(yàn)證人5.1 驗(yàn)證方案（定期指標(biāo)）PG：序號(hào)指標(biāo)網(wǎng)元驗(yàn)證頻度驗(yàn)證結(jié)果驗(yàn)證人1檢查PG的proclog，確認(rèn)各類(lèi)指令的發(fā)放成功率指標(biāo)檢查，精確到指令級(jí)別（ADD_SUB、LST_SUB等）cat /var/proclog/raw/proclog* | grep ADD_SUB | grep FAILED | wc -lcat /var/proclog/raw/proclog* | grep ADD_SUB | grep SUCC | wc lPG一次CUDB：序號(hào)指標(biāo)網(wǎng)元驗(yàn)證頻度驗(yàn)證結(jié)果驗(yàn)證人1檢查CU

13、DB狀態(tài) cudbSystemStatusCUDB一次六、應(yīng)急方案6.1 本次變更可能出現(xiàn)的問(wèn)題系統(tǒng)有可能出現(xiàn)未知的軟件錯(cuò)誤，對(duì)業(yè)務(wù)發(fā)放造成影響。6.2 應(yīng)急與倒回方案PG:使用備份的NTP配置文件倒回cp /etc/init.d/ntp.bak /etc/init.d/ntp然后重啟NTP服務(wù)service ntp restartCUDB:使用備份的NTP配置文件倒回cp /etc/sysconfig/ntp.bak /etc/sysconfig/ntp然后重啟NTP服務(wù)rcntp restart七、BIBYSITTING監(jiān)測(cè)要求PG:設(shè)備廠家網(wǎng)元類(lèi)型序號(hào)核查項(xiàng)核查及判定標(biāo)準(zhǔn)備注說(shuō)明愛(ài)立信P

14、G1dvemon status n allCAI3G ports should be E/E on all PL.MML ports should be E/E on all PL.CLI ports should be D/D on all PL.OAM ports should be E/E on all PL.Application server should be E/E on all PL.Deployment error number should be 0 on all PL.ESA processes should be UP on all the nodes.License

15、server should be UP on both SC.Block proxy should be UP on PL3.Reporter should be UP on all PL.Zookeeper should be UP on both SC and PL-3.EMA愛(ài)立信PG2fmacivealarmsThere should be no critical alarms.EMA愛(ài)立信PG15cat /var/proclog/raw/proclog* | grep ADD_SUB | grep FAILED | wc -l確保所有指令的成功率都在正常范圍EMAcat /var/p

16、roclog/raw/proclog* | grep ADD_SUB | grep SUCC | wc lCUDB:設(shè)備廠家網(wǎng)元類(lèi)型序號(hào)核查項(xiàng)核查及判定標(biāo)準(zhǔn)備注說(shuō)明愛(ài)立信CUDB1cudbSystemStatus(1) The time and date is right,(2)CUDB software version is right, (3)11B:System Monitor threads status on each system controller board are OK; 13B:System Monitor leader and BC Server status are

17、OK.(4)Clusters status on each PL and DS is OK, the memory usage lower than 80% (5) There are 4 NDBs process are running on PL Unit and 2 NDBs process are running on each DS Unit, (6) Replication Channels are OK, 'M': Master, 'Sn': Slave replication channel #n is active. (7)There is no new alarm is printed, more information refe