天天看點

園區間業務系統通路故障處理

故障現象:

客戶反映園區倉庫通路NC系統速度很慢,從園區ping NC伺服器位址時有嚴重的丢包現象。如下圖所示:

<a href="http://s3.51cto.com/wyfs02/M00/8A/8C/wKiom1gz1DKi117aAAKQFNfloAw338.png-wh_500x0-wm_3-wmp_4-s_1579186669.png" target="_blank"></a>

<a href="http://s1.51cto.com/wyfs02/M02/8A/8C/wKiom1gz1DOx2rwHAAFuqD6gfac287.png-wh_500x0-wm_3-wmp_4-s_2366122665.png" target="_blank"></a>

處理過程:

登入使用者兩個園區的核心發現,兩個園區的portchannel連接配接正常,但是其中一個園區portchannel成員端口下存在嚴重的CRC校驗錯誤,如下圖所示:

<a href="http://s1.51cto.com/wyfs02/M02/8A/89/wKioL1gz08HDzqxmAALCiT0OPh0626.png-wh_500x0-wm_3-wmp_4-s_3053852100.png" target="_blank"></a>

此時,關閉端口ten1/5/1,資料轉發從另一個端口走,關閉後再次PING NC伺服器,資料包正常

<a href="http://s3.51cto.com/wyfs02/M02/8A/89/wKioL1gz1KDgfrB5AAFB4YDNDbs447.png-wh_500x0-wm_3-wmp_4-s_1719984639.png" target="_blank"></a>

<a href="http://s3.51cto.com/wyfs02/M00/8A/8C/wKiom1gz1KCTa0kNAAFlB_3Ct-w146.png-wh_500x0-wm_3-wmp_4-s_3799085379.png" target="_blank"></a>

為進一步确認該故障,再次對ten1/5/1進行開啟操作,但打開後發現,問題立即出現

SW#ping 192.168.x.x repeat 1000 source 192.168.x.x

Type escape sequence to abort.

Sending 1000, 100-byte ICMP Echos to 192.168.x.x, timeout is 2 seconds:

Packet sent with a source address of 192.168.x.x

!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!.!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!.!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!

!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!.!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!

!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!

Success rate is 97 percent (978/1000), round-trip min/avg/max = 1/2/20 ms

SW#sh int ten1/5/1

TenGigabitEthernet1/5/1 is up, line protocol is up (connected)

  Hardware is Ten Gigabit Ethernet Port, address is 78da.6e07.7b7a (bia 78da.6e07.7b7a)

  Description: connection to laishanchangqu

  MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,

     reliability 245/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-ER

  input flow-control is on, output flow-control is on

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:03, output never, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 3209000 bits/sec, 358 packets/sec

  5 minute output rate 926000 bits/sec, 685 packets/sec

     4935914010 packets input, 5278619447870 bytes, 0 no buffer

     Received 87068 broadcasts (87050 multicasts)

     0 runts, 0 giants, 0 throttles

     277020288 input errors, 277020288 CRC, 0 frame, 0 overrun, 0 ignored

     0 input packets with dribble condition detected

     9064745454 packets output, 3287137727436 bytes, 0 underruns

     0 output errors, 0 collisions, 4 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

SW#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

SW(config)#int ten1/5/1

SW(config-if)#shut

SW(config-if)#do sh run int ten1/5/1

Building configuration...

Current configuration : 171 bytes

!

interface TenGigabitEthernet1/5/1

 switchport access vlan 133

 switchport mode access

 shutdown

 channel-group 10 mode on

end

SW(config-if)#exit

SW(config)#exit

SW#wr

Compressed configuration from 21267 bytes to 7568 bytes[OK]

Sending 1000, 100-byte ICMP Echos to 192.168.x.x timeout is 2 seconds:

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/2/20 ms

經最終确認,NC通路問題主要是由于該端口的CRC校驗錯誤引起。

解決建議:

1、檢查兩廠區之間的鍊路

2、檢查子產品

3、檢查尾纖

                                                               *********************

最終:

       第二天測試兩個園區之間的鍊路,發現一根光纖的衰減在-21左右,而該接口上的萬兆子產品可接受的衰減最小值是-15.8,後來營運商對線路進行調優檢測,最終恢複正常,測試,網絡通路正常

本文轉自 彎月樓主 51CTO部落格,原文連結:http://blog.51cto.com/05wylz/1875378,如需轉載請自行聯系原作者