IOS: EIGRP Peering Flapping, Auth Failure - %DUAL-5-NBRCHANGE: IP-EIGRP: Auth failure

This is an actual case we have raised recently with Cisco as we are having unexplained EIGRP flaps between two of our devices. It has been working for more than a year -- actually, it never had any issues when this was brought online last year.



Scenario:

EIGRP peering flaps between two devices, due to authentication failure. The output of show logging is flooded with the below syslogs repeatedly:

Nov 2 01:30:43.436 GMT: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 2: Neighbor 10.10.10.2 (GigabitEthernet1/1) is down: Auth failure


Nov 2 01:30:45.040 GMT: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 2: Neighbor 10.10.10.2 (GigabitEthernet1/1) is up: new adjacency

Nov 2 01:30:47.316 GMT: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 2: Neighbor 10.10.10.2 (GigabitEthernet1/1) is down: Auth failure

Nov 2 01:30:48.820 GMT: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 2: Neighbor 10.10.10.2 (GigabitEthernet1/1) is up: new adjacency

Topology is straightforward:

Router1 Gi1/1 <-----> Gi2/2 Router2


Router1 Gi1/1 = 10.10.10.1/24
Router2 Gi2/2 = 10.10.10.2/24

MD5 Authentication is used and the same key string is configured on both devices

Router1#show key chain MYCHAIN

 key-chain MYCHAIN
  key 1 -- text "myCiscoChain"
   accept lifetime (always valid) - (always valid) [valid now]
   send lifetime (always valid) - (always valid) [valid now]
Router1#
Router1#
Router1#
Router1# show run | begin key chain
key-chain MYCHAIN
 key 1
  key string 5 098123456SA679
...
Router1# show run int Gi1/1
interface GigabitEthernet1/1
 ip address 10.10.10.1 255.255.255.0
 ip authentication mode eigrp 2 md5
 ip authentication key-chain eigrp 2 MYCHAIN
...

Problem:
The issue was with a Level2/Severe bug with the IOS image running on one of the devices. Bug details below:

CSCdu73495 - All routes to network not seen because of invalid md5 authentication
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCdu73495

Enhanced Interior Gateway Routing Protocol (EIGRP) routes cannot be seen even when message digest algorithm 5 (MD5) is authenticated on all routers. This problem is intermittent and may occur when authentication is turned off and subsequently turned back on again. Sometimes, this problem occurs just after authentication is enabled.  

Workaround: This problem is intermittent and may be resolved by disabling and reenabling authentication a second time. This problem may automatically be resolved after a few minutes.


EIGRP Authentication problems & flaps on unrelated links


This bug is a duplicate of CSCdu73495, which causes authentication-related breakage in establishing peers, which eventually clears up on it's own after an indeterminate time. It can be triggered by bouncing peers/interfaces. You will not encounter this issue if you disable EIGRP authentication. CSCdu73495 was resolved in later versions of 12.1E IOS.

EIGRP neighbour cant be established if use MD5 authentication

C2610 EIGRP neighbour could be established via md5 authentication first time. After shut/no shut c2610 ethernet interface, it can't established any more. Via serial interface works fine.

EIGRP MD5 Authentication Breaks Neighbor Adjacencies over LANE

In a LANE environment with 3 or more devices running EIGRP, when upgrading from 12.1(6)E4 to 12.1(10)E4 on 7500's, EIGRP neighbor relationships may not be formed between devices running 12.1(10)E4. This is verified by performing a on one of the devices running 12.1(10)E4. The workaround for this scenario is to wait an unpredictable amount of time for the neighbors to converge, or remove and re-add EIGRP authentication from the interfaces on the affected devices. Also, neighbors can be statically configured in order for EIGRP to use unicast, rather than multicast.

2921-EIGRP flap due to bad TLV received on serial interface

Symptom: EIGRP flaps observed due to retransmission retry limit exceeded. Bad TLV error messages are seen in the logs. Conditions: Issue seen when 2921 replaces the 2611 device with similar configs.

Workaround: None. Apart from 2921, customer is using 2611 that works fine.


Known Affected Versions (Not a comprehensive list):
12.1(9)M

12.1(26)M
15.0M
12.1(8b)E15
12.3(12e)M
 
Fixed-In (Not comprehensive list):

12.1(10.2)M
12.2(4.2)M
12.0(30)SZ4
12.0(32)S6b
12.0(32)S7
12.0(32)SY4
12.0(32.3)S
12.1(6)E11
12.1(10.5)E
12.1(10.5)EC
12.2(4.2)PI
12.2(4.2a)DA
12.2(5.1)S
12.2(6.4)B
12.2(6.4)PB
12.2(15)BW
12.2(15)BX
12.2(15)ZN
12.0(32.11.1)SY


Workarounds:

1. Disable then re-enable EIGRP authentication;
2. Instead of MD5, use clear text authentication; or
3. Disable EIGRP authentication.

Permanent Fix:
Upgrade IOS version.

Due to intermittence/unpredictability, either use clear text authentication or disable authentication outright if IOS upgrade is not possible immediately. However, bouncing (disable/re-enable) the authentication can serve as your quick fix.

IOS: show interface FastEthernet mod/port - Detailed

The show interface output for physical interface

Router#sh interfaces FastEthernet 6/1

FastEthernet6/1 is up, line protocol is up (connected)
 Hardware is C6k 100Mb 802.3, address is 0009.11f3.8848 (bia 0009.11f3.8848)
 MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
  reliability 255/255, txload 1/255, rxload 1/255
 Encapsulation ARPA, loopback not set
 Full-duplex, 100Mb/s
 input flow-control is off, output flow-control is off
 ARP type: ARPA, ARP Timeout 04:00:00
 Last input 00:00:14, output 00:00:36, output hang never
 Last clearing of "show interface" counters never
 Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
 Queueing strategy: fifo
 Output queue :0/40 (size/max)
 5 minute input rate 0 bits/sec, 0 packets/sec
 5 minute output rate 0 bits/sec, 0 packets/sec
 1117058 packets input, 78283238 bytes, 0 no buffer
 Received 1117035 broadcasts, 0 runts, 0 giants, 0 throttles
 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
 0 watchdog, 0 multicast, 0 pause input
 0 input packets with dribble condition detected
 285811 packets output, 27449284 bytes, 0 underruns
 0 output errors, 0 collisions, 2 interface resets
 0 babbles, 0 late collision, 0 deferred
 0 lost carrier, 0 no carrier
 0 output buffer failures, 0 output buffers swapped out

Show interface output (physical interface) explained
 
up, line protocol up (connected) - the "up" is the physical layer (OSI layer 1) status of the link; the "line protocol up" is the data link layer (OSI layer 2) status of the link. Possible output are as follows:
 
up, line protocol up
up, line protocol down
down, line protocol down

Hardware - the interface hardware type, as well as the hardware/MAC address.

Description - the user-specified interface description as configured in the interface configuration mode.

MTU 1500 bytes - Maximum Transmission Unit.
BW -  Bandwidth.
DLY - Delay (in microseconds).

reliability - reliability, as fraction of 255 (where 255/255 = 100% reliability), exponential average over 5 minutes.
txload - current output load, as fraction of 255 (where 255/255 = 100% saturation), exponential average over 5 minutes.
rxload - current input load, as fraction of 255 (where 255/255 = 100% saturation), exponential average over 5 minutes.

Encapsulation - current data link/layer 2 encapsulation of the interface.
loopback - defines if loopback (hardware or software) is enabled or disabled.

Full-duplex, 100Mb/s - current duplex and speed settings of the interface.

ARP Type - the Address Resolution Protocol type enabled.
ARP Timeout - the time in hh:mm:ss for each entry remains in ARP cache before being removed.

Last input 00:00:14, output 00:00:36 - the time in hh:mm:ss when the last packet was received (input) or transmitted (output) by the interface.

output hang - the time in hh:mm:ss when the interface was reset because of a transmission that took too long.

Last clearing of "show interface" counters - the time when the interface counters are last cleared via "clear counter" command.

Input queue: 0/2000/0/0 (size/max/drops/flushes) - the input queue counters and thresholds; the first number (size) is the current number of frames in the queue; the second number (max) is the maximum number of frames in the queue before it starts dropping; the third number (drops) is the number of frames dropped because the max was exceeded; the last number (flushes) is the number of low-priority frames dropped due to Selective Packet Discard (SPD) algorithm when CPU is overloaded.

Total output drops: 0 - total number of packets dropped because the output queue is full; high output drops may indicate mismatched bandwidth settings of this and the remote connecting interface.

Queueing strategy - either First-In/First-Out (fifo), priority-list, custom-list, and weighted-fair.

Output queue :0/40 (size/max) - The number of packets in the output queue. Size is the current number of frames in the queue. Max is the number of frames the queue can hold before it starts dropping frames.

5 minute input/output rate - The average input and output rate seen by the interface in the last five minutes. The interval can be changed via the "load-interval " interface command.

packets input, bytes - Total number of error-free packets received by the system. Total number of bytes, including data and MAC encapsulation, in the error free packets received by the system.

no buffer - The number of packets received and discarded because there is no buffer space. Can be caused by broadcast storms.

Received broadcasts - The number of broadcast/multicast packets received by the interface.

runts - The number of packets that are discarded because they are smaller than the minimum packet size.

giants - The number of packets that are discarded because they exceed the maximum packet size (MTU).

throttles - The number of times the receiver on the port was disabled, possibly due to buffer or processor overload.

input errors - Includes runts, giants, no buffer, CRC, frame, overrun, and ignored counts.

CRC - Number of packets where the CRC generated by the originating far-end device does not match the checksum calculated from the data received; usually indicates noise or transmission problems on the LAN interface or the LAN bus itself.

frame - Number of packets received incorrectly having a CRC error and a noninteger number of octets A(elignment errors).

overrun - Number of times the receiver hardware was unable to hand received data to a hardware buffer because the input rate exceeded the receiver's ability to handle the data.

ignored - Number of received packets ignored by the interface because the interface hardware ran low on internal buffers.

watchdog - Number of times watchdog receive timer expired. It happens when receiving a packet with length greater than 2048.

multicast - Number of multicast packets received.

pause input - Number of times the connected device requests for a traffic pause when its receive buffer is almost full. This counter is incremented for informational purposes, since the switch accepts the frame. The pause packets stop when the connected device is able to receive the traffic.

input packets with dribble condition detected - A dribble bit error indicates that a frame is slightly too long. This frame error counter is incremented for informational purposes, since the switch accepts the frame.

packets output, bytes - Total number of error-free packets transmitted by the system. Total number of bytes, including data and MAC encapsulation, in the error free packets transmitted by the system.

underruns - Number of times that the transmitter has been run faster than the switch can handle. This can occur in a high throughput situation where an interface is hit with a high volume of bursty traffic from many other interfaces all at once. Interface resets can occur along with the underruns.

output errors - Sum of all errors that prevented the final transmission of datagrams out of the interface.

collisions - Number of times a collision occurred before the interface transmitted a frame to the media successfully. Collisions are normal for interfaces configured as half duplex but must not be seen on full duplex interfaces. If collisions increase dramatically, this points to a highly utilized link or possibly a duplex mismatch with the attached device.


interface resets - Number of times the interface transitioned from up to down to up.
 
babbles - Number of times that the transmit jabber timer expired. A jabber is a frame longer than 1518 octets (which exclude framing bits, but include FCS octets), which does not end with an even number of octets (alignment error) or has a bad FCS error.
 
late collision - Number of times a late collision occured. A late collision occurs when two devices transmit at the same time, and neither side of the connection detects a collision. The reason for this occurrence is because the time to propagate the signal from one end of the network to another is longer than the time to put the entire packet on the network. The two devices that cause the late collision never see that the other is sending until after it puts the entire packet on the network. Late collisions are not detected by the transmitter until after the first 64 byte slot time. This is because they are only detected in transmissions of packets longer than 64 bytes.



deferred - Number of frames that have been transmitted successfully after they wait because the media was busy. This is usually seen in half duplex environments where the carrier is already in use when it tries to transmit a frame.

lost carrier - The number of times the carrier was lost in transmission. This is usually caused by a bad cable. Check the physical connection on both sides.


no carrier - Number of times the carrier was not present in the transmission. This is usually caused by a bad cable. Check the physical connection on both sides.


output buffer failures, output buffers swapped out - Number of failed buffers and the number of buffers swapped out. A port buffers the packets to the Tx buffer when the rate of traffic switched to the port is high and it cannot handle the amount of traffic. The port starts to drop the packets when the Tx buffer is full and thus increases the underruns and the output buffer failure counters. The increase in the output buffer failure counters can be a sign that the ports are run at an inferior speed and/or duplex, or there is too much traffic that goes through the port.

Reference: Troubleshooting Switch Port and Interface Problems
http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a008015bfd6.shtml

IOS: %SYS-SP-3-CPUHOG: RFSS_server_action

Scenario:
A Cat6K throws the following syslog messages:

Jul 18 01:48:12.362 EDT: %SYS-SP-3-CPUHOG: Task is running for (4000)msecs, more than (2000)msecs (0/0),process = RFSS_server_action.

 

-Traceback= 4045D2CC 4045F5F8 4045F504 4047F45C 4047ED38 4047F31C 40481F5C 40489F04 4048A3CC 4048AF5C 40485DE4 4048B1AC 404816A8 402E41D8 40451534 4029A764

 

Jul 18 01:48:14.366 EDT: %SYS-SP-3-CPUHOG: Task is running for (2000)msecs, more than (2000)msecs (1/0),process = RFSS_server_action.

 

-Traceback= 4045D2A8 4045F5F8 4045F504 4047F45C 4047ED38 4047F31C 40481F5C 40489F04 4048A3CC 4048AF5C 40485DE4 4048B1AC 404816A8 402E41D8 40451534 4029A764

 

Jul 18 01:48:18.370 EDT: %SYS-SP-3-CPUHOG: Task is running for (2000)msecs, more than (2000)msecs (2/1),process = RFSS_server_action.

 

-Traceback= 4045D2A8 4045F5F8 4045F504 4047F45C 4047ED38 4047F31C 40481F5C 40489F04 4048A3CC 4048AF5C 40485DE4 4048B1AC 4048B504 404817C0 402E440C 40451660

Description:

The traceback shown indicates a problem with writing into the flash disk. Running the privileged-mode command "dir disk1:" will cause your login session to apparently hang for a few minutes. After that the logs will be filled up with a new batch of the above %SYS-SP-3-CPUHOG syslogs and traceback messages.
 

------------------ show disk1: all ------------------
172683264 bytes available (83296256 bytes used)
******** ATA Flash Card Geometry/Format Info ********
ATA CARD GEOMETRY
 Number of Heads: 16
 Number of Cylinders 978
 Sectors per Cylinder 32
 Sector Size 512
 Total Sectors 500736
 
ATA CARD FORMAT
 Number of FAT Sectors 245
 Sectors Per Cluster 8
 Number of Clusters 62495
 Number of Data Sectors 500596
 Base Root Sector 598
 Base FAT Sector 108
 Base Data Sector 630 %
 
Error show disk1: (TF I/O failed in data-in phase)

 
Workaround/Resolution:

1. Reseat the compact flash card.
2. If error still occurs, reformat the flash card.
3. If error still occurs, replace the flash card.

IOS: %EARL_L3_ASIC-SP-3-INTR_WARN: EARL L3 ASIC: Non-fatal interrupt Packet Parser block interrupt

Dec 18 09:54:43.989 JST: %EARL_L3_ASIC-SP-STDBY-3-INTR_WARN: EARL L3 ASIC: Non-fatal interrupt Packet Parser block interrupt
Dec 18 09:54:43.993 JST: %EARL_L3_ASIC-SP-3-INTR_WARN: EARL L3 ASIC: Non-fatal interrupt Packet Parser block interrupt

Description
These messages are indicating that the switch has received an invalid packet which contained a Layer 3 IP checksum error. These packets are normally being dropped silently within older IOS. In some IOS releases, the switch informs of this condition to warn users that there is (are) devices outside sending IP packets with checksum errors and/or with wrong length.

See CSCdz10360 (Need a CLI to be able to disable L3 error checking in HW) regarding this enhancement.

Workaround
These messages are purely informational. You may either:

1. SPAN all the Vlans and look at layer3 IP source address then remove the device generating invalid packets (unfortunately the switch doesn't track the IP address. The only way is to sniff every suspected Vlan to find out where those invalid packets are coming from).



2. Configure (this is a new config option added by means of CSCdz10360):
   no mls verify ip checksum ---> to stop to check for packet checksum errors
   no mls verify ip length ---> to stop to check for packet length errors
   no mls verify ip length minimum ---> to eliminate check for IP packets that are minimum length.
   no mls verify ip same-address ---> to stop checking for packet having equal source and destination IP address.



3. Do nothing as these are pure informational.

IOS: %ETHCNTR-3-LOOP_BACK_DETECTED : Keepalive packet loop-back detected on [chars]

Scenario
The switch reports this error message, and the port is forced to linkdown:
%ETHCNTR-3-LOOP_BACK_DETECTED : Keepalive packet loop-back detected on [chars]

Oct 2 10:40:13: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected on GigabitEthernet0/1
Oct 2 10:40:13: %PM-4-ERR_DISABLE: loopback error detected on Gi0/1, putting Gi0/1 in err-disable state

Description
The problem occurs because the keepalive packet is looped back to the port that sent the keepalive. Keepalives are sent on the Catalyst switches in order to prevent loops in the network. Keepalives are enabled by default on all interfaces. You see this problem on the device that detects and breaks the loop, but not on the device that causes the loop.

Workaround
Issue the no keepalive interface command in order to disable keepalives. A disablement of the keepalive prevents errdisablement of the interface, but it does not remove the loop.

Permanent Fix
In Cisco IOS Software Release 12.2(x)SE-based releases and later, keepalives are not sent on fiber and uplink interfaces by default. Upgrading the IOS version to this or later images should prevent the above issue in the first place.

IOS: IP SLA : SNMP : Router crashes and reloads if up for more than 497 days

CSCsa57468
rttmon-mib does not return getnext value when queried via snmp

Symptom:
Concord poller crashes when polling a router that has been configured with IP SLA. Infact this DDTS will surface when doing snmp gets for the objects mentioned in the Conditions section below coming from any NMS (e.g. Concord, IPM, Spectrum, etc.)

Conditions:
The SNMP GETNEXT request is sent to the router for the following OIDs:

• rttMonJitterStatsCompletions
• rttMonStatsCaptureCompletions
• rttMonStatsTotalsInitiations
• rttMonStatsCaptureEntry (rttMonStatsCaptureCompletion etc.)
• rttMonStatsCollectEntry
• rttMonStatsTotalsEntry
• rttMonJitterStatsEntry
• rttMonHTTPStatsEntry.

The router does not return the next index of these OIDs, but the same index. This happens only when the router has been up and running for longer than 497 days.

Affected IOS Versions:

• 12.2(15)T
• 12.2SXH

Workaround:
This problem is only happening when polling the CISCO-RTTMON-MIB via snmp get. Use the IOS CLI to avoid it.

Permanent Fix:
Upgrade the IOS version.

Fixed in:

• 12.3(14.12)M
  
• 12.4(1.5)M
  
• 12.2(33)SRC
  
• 12.2(40)SE
  
• 12.2(44)SE
  
• 12.3(11)T6
  
• 12.3(11)YW
  
• 12.3(14)T2
  
• 12.4(1.8)T
  
• 12.4(1a)M
  
• 12.2(33)SXI
  
• 12.2(32.8.80)SR
  
• 12.2(32.8.11)XID112.9
  
• 12.2(33.1.7)SXH
  
• 12.2(33)SXH2
  
• 12.2(33)SB
  
• 12.2(32.8.99a)SR133
  
• 12.2(32.8.11)XJC153.1

IOS: %SSH-3-PRIVATEKEY: Unable to retrieve RSA private key

Symptoms:
The device getting numerous %SSH-3-PRIVATEKEY syslogs, usually followed by a traceback such as the following:

Nov 7 02:40:49.542 GMT: %SSH-3-PRIVATEKEY: Unable to retrieve RSA private key for
-Process= "SSH Process", ipl= 0, pid= 148
-Traceback= 61D48360 61D44B24 61D462C4 6053BD88 6053BD6C
Nov 8 02:16:22.452 GMT: %SSH-3-PRIVATEKEY: Unable to retrieve RSA private key for
-Process= "SSH Process", ipl= 0, pid= 148
-Traceback= 61D48360 61D44B24 61D462C4 6053BD88 6053BD6C

Explanation:
Often seen if hostname or domain name of the router has been changed.

Workaround/Fix:

• Remove existing RSA Key:
  crypto key zeroize rsa
  
• Gnerate RSA key with the following commands:
  
  show crypto key mypubkey rsa
  crypto key gen rsa general-keys label label
  ip ssh rsa keypair-name label
  
  where label = unique label/identifier

Basic MQC Configuration

Three easy steps to MQC (Modular QoS CLI) Configuration:
Step 1: Classify traffic via class-map
Step 2: Assign policies to the traffic classes via policy-map
Step 3: Apply above policies to an interface via service-policy

! ------------------------------------
! Sample Scenraio:
! ------------------------------------
For the following traffic going out Serial0/1 of the device, do the following:
* for voice traffic, reserve 256kbps priority bandwidth
* for email traffic (pop3, imap, smtp), reserve 128kbps bandwith
* for telnet traffic coming from 10.10.10.10, limit to 3200bps bandwidth

! ------------------------------------
! BEGIN CONFIGURATION
! ------------------------------------
Router(config)#access-list 101 host 10.10.10.10 any eq 23
Router(config)#class-map VOICE
Router(config-cmap)#match protocol rtp
Router(config-cmap)#exit
Router(config)#class-map match-any EMAIL
Router(config-cmap)#match protocol pop3
Router(config-cmap)#match protocol imap
Router(config-cmap)#match protocol smtp
Router(config-cmap)#exit
Router(config)#class-map ACL_101
Router(config-cmap)#match access-group 101
Router(config-cmap)#exit

Router(config)#policy-map MY_POLICY
Router(config-pmap)#class VOICE
Router(config-pmap-c)#priority 256
Router(config-pmap-c)#exit
Router(config-pmap)#class EMAIL
Router(config-pmap-c)#bandwidth 128
Router(config-pmap-c)#exit
Router(config-pmap)#class ACL_101
Router(config-pmap-c)#police 3200
Router(config-pmap-c)#exit
Router(config-pmap)#exit

Router(config)#interface Serial0/1
Router(config-if)#service-policy output MY_POLICY
Router(config-if)#exit
Router(config)#

! ------------------------------------
! NOTES
! ------------------------------------

Router(config)#class-map [match-all|match-any] class_name

match-all - the class must match all the succeeding criteria

match-any - the class must match any of the succeeding criteria

if not specified, defaults to match-all

Router(config-cmap)#match {protocol|access-group} value

protocol - based on known traffic classes via NBAR

access-group - based on ACLs

not limited to the above criteria; other criteria include class-map (i.e. nested class-maps), CoS, DSCP, IP Precedence, input-interface, MAC address, QoS group, UDP Port Ranges

Router(config-if)#service-policy {input|output} policy-name

only one policy per direction per interface can be applied;

that is, each interface can have at most one inbound policy and one outbound policy.

Other command syntax will be dealt with in another post.

IOS: %BGP_MPLS-3-GEN_ERROR

Mar 18 20:41:38.892 EDT: %BGP_MPLS-3-GEN_ERROR: BGP: MPLS outlabel changed, MPLS forw not updated, prefix not in routing table -Traceback= 10D36950 10D3709C 10B10388 10B10718 10AEEFD0 10AEF030 10B53A50 10B53DC0 10AF588C 10AFD610 10AFE8E0 10A44524 10A3B6D4
Mar 18 20:41:38.892 EDT: %BGP_MPLS-3-GEN_ERROR: BGP: MPLS outlabel changed, MPLS forw not updated, prefix not in routing table -Traceback= 10D36950 10D3709C 10B10388 10B10718 10AEEFD0 10AEF030 10B53A50 10B53DC0 10AF588C 10AFD610 10AFE8E0 10A44524 10A3B6D4
Mar 18 20:41:38.892 EDT: %BGP_MPLS-3-GEN_ERROR: BGP: MPLS outlabel changed, MPLS forw not updated, prefix not in routing table -Traceback= 10D36950 10D3709C 10B10388 10B10718 10AEEFD0 10AEF030 10B53A50 10B53DC0 10AF588C 10AFD610 10AFE8E0 10A44524 10A3B6D4


Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-ENTSERVICES-M), Version
12.2(50)SG1, RELEASE SOFTWARE (fc2)
Technical Support:
http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Tue 10-Feb-09 00:17 by prod_rel_team
Image text-base: 0x10000000, data-base: 0x124FED8C

ROM: 12.2(44r)SG
Darkside Revision 0, Jawa Revision 11, Tatooine Revision 140, Forerunner Revision 1.74

MyRouter uptime is 5 days, 3 hours, 12 minutes
System returned to ROM by power-on
System restarted at 19:50:40 EDT Fri Mar 13 2009
System image file is "bootflash:/cat4500e-entservices-mz.122-50.SG1.bin"

cisco WS-C4900M (MPC8548) processor (revision 2) with 524288K bytes of memory.
Processor board ID JAE130628BD
MPC8548 CPU at 1.33GHz, Cisco Catalyst 4900M
Last reset from PowerUp
1 Virtual Ethernet interface
36 Gigabit Ethernet interfaces
16 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

Configuration register is 0x2102

---

CSCse15707: Trace back seen at bgp_ipv4_mpls_label_change.

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCse15707

First Found-In:

12.2(32.8.7)SRB

12.2(32.8.8)SRA

12.4(12.15)PI7e

12.2(31)SB9

12.2(31.4.5)SB11

12.2(31.4.11)SB12

12.2(28.5.24)SB13

Fixed-In:

12.2(32.8.63)SR

12.2(33)SRC

12.4(13.8)PI7c

12.2(33)SRB3

12.2(33.2.18)SRB

12.2(33)SB

12.2(32.8.99a)SR133

12.2(31)SB13

12.2(32.8.1)YCA172.24

12.4(21.14.9)PIC1

Symptoms: A router may generate the following error message and a traceback:

%BGP_MPLS-3-GEN_ERROR: BGP: MPLS outlabel changed, MPLS forw not updated, prefix not in routing table

Conditions: This symptom is observed on a Cisco router that functions in a VPN carrier supporting carrier topology and that is configured for BGP and IPv4.

Workaround: This is a cosmetic issue, the traceback is harmless and the functionality of the router is not affected.

SNMP ifIndex persistence

Your Ad Here

Your network monitoring application, such as MRTG, is configured to monitor a specific interface. Since there are more than one interface on a device, monitoring will be based on the specific interface's assigned ifIndex.

However, after a hard device reboot, the ifIndex has changed, causing problems with MRTG reporting. This may occur intermittently; sometimes, ifIndex are the same after a reboot, sometimes they change (especially after adding logical interfaces and/or FastEthernet or GigabitEthernet modules).

This has been addressed in RFC2863 (updatint the ifIndex definition under RFC1213). As such, Cisco introduced SNMP ifIndex persistence; simply put, the current ifIndex assigned to the interface will be the permanent.

Configuration:

SNMP ifIndex persistence can be applied globally to all interfaces, or only to specific ones. The syntax are as follows:

 MyDevice(config)# snmp-server ifindex persist
 MyDevice(config-if)# snmp ifindex persist

Note: The actual syntax may vary depending on the IOS version; older versions use snmp-server ifindex persist instead of just snmp ifindex persist.

After that, you need to save the configuration via write mem or copy run start.

Verification

To retain the ifIndex values, like the configuration and vlan information in some older devices, it will be saved in the nvram. Check the NVRAM for a file called ifIndex-table.

MyDevice#dir nvram:ifIndex-table
Directory of nvram:/ifIndex-table

   3 -rw- 6712 <no date> ifIndex-table

1964024 bytes total (1752173 bytes free)
MyDevice#


Once that is configured, the changing ifIndex problem should be resolved and reporting tools such as MRTG will have no problems monitoring the correct interface.

References:
Interface Index (ifIndex) Persistence

TCP/UDP Ports for XBOX360

As you know, XBOX360 is one of the latest console gaming platforms available with network capability. Suppose you are network security consultant and you want restrict the ports used in the subnet used by your XBOX360 devices, which ports are you going to open?

Here are the required ports for the XBOX360 to properly connect to the network/Internet:

Outgoing ports:	Description
UDP 53	DNS queries
UDP 88	Kerberos
UDP/TCP 3074	XBox live
UDP 6500	Peer queries
UDP 13139	Peer ping
UDP 22042	Server queries
TCP/UDP 22043-22050	Voice chat

Incoming ports:	Description
UDP/TCP 3074	Xbox live

Given the above information, you can then create the access-lists as necessary.


EXAMPLE with Cat6000 as the gateway:
Below is a snippet of an actual access-list configured on a real-live Cat6000 device serving as the gateway for subnet 10.10.180.0/25:

ip access-list extended SUBNET_XBOX_IN
 remark <<Permit outbound XBOX360 ports>>
 permit udp 10.10.180.0 0.0.0.127 any eq 88
 permit tcp 10.10.180.0 0.0.0.127 any eq 3074
 permit udp 10.10.180.0 0.0.0.127 any eq 3074
 permit udp 10.10.180.0 0.0.0.127 any eq 6500
 permit udp 10.10.180.0 0.0.0.127 any eq 13139
 permit udp 10.10.180.0 0.0.0.127 any range 22042 22050
 permit tcp 10.10.180.0 0.0.0.127 any range 22043 22050
 deny ip any any

...

ip access-list extended SUBNET_XBOX_OUT
 remark <<Permit inbound XBOX360 ports>>
 permit tcp 10.10.180.0 0.0.0.127 any eq 3074
 permit udp 10.10.180.0 0.0.0.127 any eq 3074
 deny ip any any

...

interface Vlan180
 description "XBOX360 Subnet"
 ip address 10.10.180.254 255.255.255.128
 ip access-group SUBNET_XBOX_IN in
 ip access-group SUBNET_XBOX_OUT out
...


Note that in the above examples, outbound XBOX ports are in the inbound switch access-list; likewise, inbound XBOX ports are in the outbound switch access-list. This is with respect to the point of view of the Cat6000 where the ACLs are defined. The XBox "outbound" ports go out from the Xbox into the switch, hence we use the inbound access-list. Similarly, Xbox inbound ports come from the switch out to the XBox subnet.

Of course, the access-lists would be slightly different if the subnet's default gateway is, say, a PIX/ASA, or a checkpoint firewall, or a CatOS switch. Regardless, the same requirements apply; the above listed ports should be permitted.

VPN: Clearing IPSEC Tunnels

As per the IPSEC Checklist and Best Practices, whenever changes are going to be done to live IPSEC tunnels, it is a good practice to turn off the tunnels.

Cisco IOS Router:
clear crypto isakmp []
clear crypto sa entry

Cisco PIX 6.X / 7.X:
clear crypto isakmp sa
clear crypto ipsec sa

VPN Concentrator 3000:
Administration --> Administer Sessions --> Logout link of tunnel

IOS: %CRYPTO-4-PKT_REPLAY_ERR replay check failed

Your Ad Here

%CRYPTO-4-PKT_REPLAY_ERR : [chars] connection id=[dec]

---

IOS 12.4 --> Syslogs --> CRYPTO Messages
http://www.cisco.com/en/US/products/ps6350/products_system_message_guide_chapter09186a0080462676.html#wp164939
Error Message:
%CRYPTO-4-PKT_REPLAY_ERR : [chars] connection id=[dec]

Explanation: The replay processing has failed. The failed replay processing may be a temporarycondition caused by the wait for new SAs to be established. In the inbound case, this error might also be caused by an actual replay attack. This activity can be considered a hostile event.

Recommended Action: If the problem appears to be more than a transient one, contact the peer administrator.

---

CSCeg43855 - Router generated traffic causes anti-replay errors
http://www.cisco.com/cgi-bin/Support/Bugtool/onebug.pl?bugid=CSCeg43855&Submit=Search

Symptoms: An encrypting router may send traffic that is locally originated (such as keepalive packets or routing update packets) out of order after the packets have been encrypted. Because of the anti-replay check failure, these packets are dropped on the receiving router.

Conditions: This symptom is observed when a multipoint GRE (mGRE) and IPSec tunnel is built between two routers.

Workaround: Turn off packet authentication for the configured IPSec transform.

Further Problem Description: On a Cisco 7200 series that functions as the receiving router, you can observe the symptom in the output of the show crypto ipsec sa detail or show pas isa interface command.

---

IOS 12.4 --> IPSec Anti-Replay Window: Expanding and Disabling
http://www.cisco.com/en/US/products/ps6350/products_configuration_guide_chapter09186a0080455ad4.html
Troubleshooting Tips:
If your replay window size has not been set to a number that is high enough for the number of packets received, you will receive a system message such as the following:

*Nov 17 19:27:32.279: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed connection id=1

The above message is generated when a received packet is judged to be outside the anti-replay window.

---

Additional Notes:

If there's no interruption of service, it could just be a normal and temporary condition, especially if the SAs (IPSEC tunnels) are still being established.

Otherwise, I suggest setting the anti-replay window to, say, 1024.

crypto ipsec security-association replay window-size 1024

Take note that the above command is introduced in 12.3(14)T; older versions do not support this command.

IOS: %HW_VPN-1-HPRXERR Packet Encryption/Decryption Errors

%HW_VPN-1-HPRXERR : [chars]: Packet Encryption/Decryption error, status=[int]

---

CSCdt40220 - AIM encryption produces Packet Encryption/Decryption error
http://www.cisco.com/cgi-bin/Support/Bugtool/onebug.pl?bugid=CSCdt40220&Submit=Search

Symptoms:
A router displays one of the following error messages:

HW_VPN-1-HPRXERR: Hardware VPN0/2: Packet Encryption/Decryption error, status=4612

This is a notification message seen on the console of the DECRYPTING PEER that tells the user that IPSEC packets have been received out of order. Re-ordering can occur in one of 3 places:
1. encrypting peer
2. network
3. decrypting peer

Only in rare cases can this occur in the decrypting peer.

The only known way for this to occur in the decrypting peer is for a packet to be bumped to process switch while the following packets from the same tunnel are fast or cef switched. This could happen if the packet is fragmented and needs re-assembly.

The following lists some of the common scenarios that might introduce out-of-order IPSEC packets. These scenrios are considered normal behaviors:

1. Fragmentation - the decrypting peer uses process switching to fragmented packets. To minimize the impact of this, Look-Ahead-Fragmentation should be enabled. This feature was added to IOS via CSCdw77514.

2. QoS: QoS scheduling mechanism happening after IPSec encryption could cause packets in the same IPsec SAs to be transmitted out-of-order.

3. Pak_priority: pak_priority is an internal flag set by the IOS to some of the router generated packets that are considered critical, e.g., routing updates, interface keepalives. When output interface queue is congested, router will honor the pak_priority flags to make sure the high priority packets are transmitted first. So in the GRE over IPsec and dynamic routing protocol design, the ESP packets could become out-of-order if the egress interface is congested and the router has to transmit the encrypted routing update first.

Conditions:
Either of the messages may be displayed depending on whether Authentication Header (AH) or Encapsulation Protocol (ESP) encapsulation is used. In addition, the ah_seq_fail or esp_seq_fail error counts increment in the output of the show crypto engine accelerator statistic privileged EXEC command.

Workaround:
- Set the maximum transmission unit (MTU) size of inbound streams to less than 1400 bytes.
- enable Look-Ahead-Fragmentation

---

WORKAROUND:

1. Adjust the interface MTU (preferably below 1400):

interface type mod/port
ip mtu byte


2. Adjust Fragmentation (See Pre-Fragmentation for IPSEC VPNs):

crypto ipsec df-bit clear
crypto ipsec fragmentation before-encryption

-- OR --

crypto ipsec df-bit clear
interface type mod/port
crypto ipsec fragmentation before-encryption

---

REFERENCES:

Pre-fragmentation for IPSec VPNs
http://www.cisco.com/en/US/products/sw/iosswrel/ps1839/products_feature_guide09186a0080115533.html