|
|
Friday August 11, 2006 at 4:45 IBS has a core router fail down hard. Our
monitoring system work flawlessly sending pages to noc personnel alerting of the
failure. The best part was Impact only received 4 calls from T-1 customers
directly connected to the 7200 VXR router that failed. I was the first to
respond to the outage. I could not physically see the router lights from my
position in the noc but I could see the lights on the T-1 CSU/DSU all off, not
good. So I stood up in my chair to see the lights on the router, zero – none
total lights out. Wow, very weird the router has redundant power supplies
connected to separate electrical circuits. First thought power so I glanced to
some other equipment attached to the power, fine. Ok reboot the router, I walked
behind the rack rows to the router and cycled the power on each of the power
supplies, lights for only one second, wait a minute I smell burning electronics.
I grabbed my camera phone quick and snapped a picture (I know those things were
good for something) of the connections, noted the DS3 connections and removed
all the connections, and pulled the router. Ok I know the router is having
problems so I start at the first logical place the NPE (Network Processing
Engine), still nothing. Ok lets use what the man upstairs gave me, pulled all
cards, and physically inspect all cards, perfect. Now take a look an the 7204
chassis looks good but there is a odd smell from the fan assembly. Ok found it
the fan chassis burned up and caused the failure. I would like to take a moment
and say that every Cisco guru I have spoke to says the chassis malfunction is
unheard of. Dam why can’t I win the lottery instead!!!
The Fix
The Point
The Configuration
CORE Router:
BACKUP Router:
Show commands :
