Tuesday, December 1, 2009

Fortigate: Password recovery

To reset the FortiGate unit password:
  1. Connect the terminal to the FortiGate unit using the null modem cable.
  2. Log on at the console with the user name "maintainer" and password "bcpb" followed immediately by the unit serial number. You must enter the alphabetic characters of the serial number in upper case.
  3. Enter the following commands:
    config system admin
    edit admin
    set password mypassword
    end

Special Notes:
  • You must first power down the FortiGate unit, and power it up again.
  • Follow the above steps within one minute of the restart

If the maintainer login is initially unsuccessful, try the following two tips to ensure successful login:
  • You many not have the correct serial #. Copy the serial # displayed on the console during initial boot-up and paste it into a terminal editor window.
  • In the terminal editor window, finish composing the full password by adding "bcpb" before the serial # and then copy & paste the entire password into the console.

Wednesday, November 25, 2009

F5 BIGIP: Verify/Restart SNMP Daemon

Just in case you need to check the status and/or restart the SNMP daemon of the bigip (i.e., because it has stopped responding to SNMP polling), enter the following commands via the CLI:

For BIGIP v4
  1. Check the SNMP daemon status:
    /etc/bigstart/status/S40snmpd status

    The correct output should be:
    Status snmpd: (pid xxxxx) is running
    Status bigsnmpd: (pid yyyyy) is running
    Status rlxsnmpd: is not running


  2. If the result is different from above (i.e., bigsnmpd is not running), restart the SNMP daemon:
    /etc/bigstart/status/S40snmpd restart



For BIGIP v9
  1. Check the current status of the SNMP daemon:
    bigstart status snmpd


  2. Restart the SNMP daemon
    bigstart restart snmpd


  3. Verify status of the SNMP daemon:
    bigstart status snmpd


  4. Example:
    [root@bigip:Active]~# bigstart status snmpd
    snmpd run (pid 12707) 90 days, 1 start
    [root@bigip:Active]~# bigstart restart snmpd
    [root@bigip:Active]~# bigstart status snmpd
    snmpd run (pid 4822) 6 seconds, 2 starts
    [root@bigip:Active]~#

Wednesday, November 11, 2009

CatOS : SYS-2-MOD_TEMPSENSORFAIL flood from X6148A-GE-45AF

CSCsl37513
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsl37513

SYS-2-MOD_TEMPSENSORFAIL:Module w/ X6148A-GE-45AF and CatOS

Symptom:
Numerous WS-X6148 linecards generate the following error:
%SYS-2-MOD_TEMPSENSORFAIL:Module # temperature sensors failed, please %powercycle the module


Conditions:
No production impact related to this message.

Workaround:
Powercycle module as requested by the error message.
  • set module power down module_number
  • set module power up module_number

Permanent Fix:
Upgrade IOS/CatOS to the below versions or later:
8.7(0.22)FW124
8.7(1.62)LAR
8.6(5.7)
8.7(0.22)BUB48
12.2(33.3.13)SXH
12.2(33)SXH4

IOS: IP SLA : SNMP : Router crashes and reloads if up for more than 497 days

CSCsa57468
rttmon-mib does not return getnext value when queried via snmp


Symptom:
Concord poller crashes when polling a router that has been configured with IP SLA. Infact this DDTS will surface when doing snmp gets for the objects mentioned in the Conditions section below coming from any NMS (e.g. Concord, IPM, Spectrum, etc.)

Conditions:
The SNMP GETNEXT request is sent to the router for the following OIDs:
  • rttMonJitterStatsCompletions
  • rttMonStatsCaptureCompletions
  • rttMonStatsTotalsInitiations
  • rttMonStatsCaptureEntry (rttMonStatsCaptureCompletion etc.)
  • rttMonStatsCollectEntry
  • rttMonStatsTotalsEntry
  • rttMonJitterStatsEntry
  • rttMonHTTPStatsEntry.
The router does not return the next index of these OIDs, but the same index. This happens only when the router has been up and running for longer than 497 days.

Affected IOS Versions:
  • 12.2(15)T
  • 12.2SXH

Workaround:
This problem is only happening when polling the CISCO-RTTMON-MIB via snmp get. Use the IOS CLI to avoid it.

Permanent Fix:
Upgrade the IOS version.

Fixed in:
  • 12.3(14.12)M
  • 12.4(1.5)M
  • 12.2(33)SRC
  • 12.2(40)SE
  • 12.2(44)SE
  • 12.3(11)T6
  • 12.3(11)YW
  • 12.3(14)T2
  • 12.4(1.8)T
  • 12.4(1a)M
  • 12.2(33)SXI
  • 12.2(32.8.80)SR
  • 12.2(32.8.11)XID112.9
  • 12.2(33.1.7)SXH
  • 12.2(33)SXH2
  • 12.2(33)SB
  • 12.2(32.8.99a)SR133
  • 12.2(32.8.11)XJC153.1

Sunday, November 8, 2009

IOS: %SSH-3-PRIVATEKEY: Unable to retrieve RSA private key

Symptoms:
The device getting numerous %SSH-3-PRIVATEKEY syslogs, usually followed by a traceback such as the following:

Nov 7 02:40:49.542 GMT: %SSH-3-PRIVATEKEY: Unable to retrieve RSA private key for
-Process= "SSH Process", ipl= 0, pid= 148
-Traceback= 61D48360 61D44B24 61D462C4 6053BD88 6053BD6C
Nov 8 02:16:22.452 GMT: %SSH-3-PRIVATEKEY: Unable to retrieve RSA private key for
-Process= "SSH Process", ipl= 0, pid= 148
-Traceback= 61D48360 61D44B24 61D462C4 6053BD88 6053BD6C


Explanation:
Often seen if hostname or domain name of the router has been changed.

Workaround/Fix:

  • Remove existing RSA Key:
    crypto key zeroize rsa
  • Gnerate RSA key with the following commands:

    show crypto key mypubkey rsa
    crypto key gen rsa general-keys label label
    ip ssh rsa keypair-name label

    where label = unique label/identifier

Wednesday, November 4, 2009

Wireless: %DTL-1-ARP_POISON_DETECTED

CSCsm25943 Change label for %DTL-1-ARP_POISON_DETECTED message
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsm25943

Symptom:

A Wireless LAN Controller may emit a message similar to the following:

DTL-1-ARP_POISON_DETECTED: STA [00:01:02:0e:54:c4, 0.0.0.0] ARP (op 1) received with invalid SPA 192.168.1.152/TPA 192.168.0.206

However, when one peruses the entry in the Cisco Wireless LAN Controller System Message Guide, 4.2, for this message, he may find it to be misleading and bereft of useful information.

Conditions:

This message does not necessarily imply that any actual "ARP poisoning" (ARP spoofing) is going on. Rather, it is emitted whenever the following conditions pertain:

- WLAN is configured with DHCP Required
- A client, after associating on that WLAN, transmits an ARP message without first DHCPing

This may be normal behavior - for example, when the client is statically addressed, or when the client is holding a valid DHCP lease from a prior association.

The effect of this condition is that the client will be unable to send or receive any data traffic, until it DHCPs thru the WLC.

In more detail, here is how to interpret the example message above:

DTL-1-ARP_POISON_DETECTED: STA [00:01:02:0e:54:c4, 0.0.0.0] ARP (op 1) received with invalid SPA 192.168.1.152/TPA 192.168.0.206

DTL-1-ARP_POISON_DETECTED
- WLC received an ARP packet from a client in DHCP_REQ state

STA [00:01:02:0e:54:c4, 0.0.0.0]
- the client ("STA" - 802.11 wireless station) has a MAC address of 00:01:02:0e:54:c4, and an IP address unknown to the WLC ("0.0.0.0")

ARP (op 1)
- the offending packet received from client was an ARP request (opcode 1)

invalid SPA 192.168.1.152/TPA 192.168.0.206
- the source IP address (SPA - "sender protocol address") of the ARP request was 192.168.1.152
- the target IP address (TPA - "target protocol address") of the ARP request was 192.168.0.206

Workaround:

  1. figure out whether or not you want to force your wireless clients to DHCP first, after associating, before they can send IP packets.


  2. If no, then unconfigure DHCP required, and you won't get this problem.


  3. If yes, then configure all clients to use DHCP.


  4. If the client is configured for DHCP, but still sometimes sends IP packets after associating without re-DHCPing, then:

    • See if the client eventually does re-DHCP & if so doesn't suffer an unacceptable outage before re-DHCPing. If the outage before re-DHCPing is acceptable, then you can just ignore this message.


    • If the client never does re-DHCP after associating, then it will never be able to pass L3 traffic. So in that case, either figure out how to change the client's behavior so that it always does re-DHCP after associating, or else just accept that this client won't work in this application, or else reconsider your decision to use "DHCP required".



Further Problem Description:

If the source IP address (SPA) of the ARP is an APIPA address (i.e. one in 169.254.0.0 /16), then this may be indicative of the STA's attempting but failing to acquire an address via DHCP. In which case you may want to verify that your DHCP implementation works.

1st Found-In:
4.2(61.0)

Fixed-In:
7.0(63.0)

Wireless: %APF-3-USER_DEL_FAILED

Event : %APF-3-USER_DEL_FAILED: Unable to delete username unknown for mobile mac-address

Explanation: This error can mean slightly different things depending on EAP method. Basically it is a side effect of an EAP method with identity protection.

EAP authentication is done in two phases. The first phase of authentication uses generic anonymous external identity in order to establish the tunnel. In phase 2, client authentication is done in the established tunnel. The client sends the original username and password to authenticate and establish a client authorization policy. As this authentication method hides the original user name at the first phase of authentication, the controller does not have a way to add the correct username to the authenticated user list. So the controller uses the anonymous username. The end result generates this error.

Further details on the related bug below:



%APF-1-USER_DEL_FAILED: apf_ms.c:5055 flooding msglogs.
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsz51403

Symptom:
The "%APF-1-USER_DEL_FAILED: apf_ms.c:5055" message floods msglogs

Conditions:
1. Multiple clients connect to the controller with the same user name, or
2. AAA server returns a user name that is different to what is registered by the client.

Workaround:
No, but it does not affect any controller feature

1st Found-In
  • 5.2(178.12)
  • 5.2(178.13)

    Fixed-In
  • 6.0(176.0)
  • 5.2(186.0)
  • 6.1(34.0)
  • 6.0(182.0)
  • 4.2(205.1)
  • 5.2(193.0)
  • 4.2(207.0)
  • Tuesday, July 28, 2009

    BIG-IP License Error - Permission denied

    Scenario:
    After a reboot, the BIG-IP returns a licensing error. Reactivating the license does not work as well.

    In the qkview output, we see the following:

    2009-07-19 22:56:17,428 ERROR [Thread-15] util.F5Error: - An error has occurred while trying to process your request.

    java.io.FileNotFoundException: /var/tmp/bigip.license (Permission denied)



    Workaround:
  • Delete /var/tmp/bigip.license file if it exists.

  • Ensure that the user you are logged into has full premissions / full admin rights to the BIG-IP box.

  • Reactivate the license file.
  • Sunday, June 7, 2009

    Basic MQC Configuration

    Three easy steps to MQC (Modular QoS CLI) Configuration:
    Step 1: Classify traffic via class-map
    Step 2: Assign policies to the traffic classes via policy-map
    Step 3: Apply above policies to an interface via service-policy

    ! ------------------------------------
    ! Sample Scenraio:
    ! ------------------------------------
    For the following traffic going out Serial0/1 of the device, do the following:
    * for voice traffic, reserve 256kbps priority bandwidth
    * for email traffic (pop3, imap, smtp), reserve 128kbps bandwith
    * for telnet traffic coming from 10.10.10.10, limit to 3200bps bandwidth

    ! ------------------------------------
    ! BEGIN CONFIGURATION
    ! ------------------------------------
    Router(config)#access-list 101 host 10.10.10.10 any eq 23
    Router(config)#class-map VOICE
    Router(config-cmap)#match protocol rtp
    Router(config-cmap)#exit
    Router(config)#class-map match-any EMAIL
    Router(config-cmap)#match protocol pop3
    Router(config-cmap)#match protocol imap
    Router(config-cmap)#match protocol smtp
    Router(config-cmap)#exit
    Router(config)#class-map ACL_101
    Router(config-cmap)#match access-group 101
    Router(config-cmap)#exit

    Router(config)#policy-map MY_POLICY
    Router(config-pmap)#class VOICE
    Router(config-pmap-c)#priority 256
    Router(config-pmap-c)#exit
    Router(config-pmap)#class EMAIL
    Router(config-pmap-c)#bandwidth 128
    Router(config-pmap-c)#exit
    Router(config-pmap)#class ACL_101
    Router(config-pmap-c)#police 3200
    Router(config-pmap-c)#exit
    Router(config-pmap)#exit

    Router(config)#interface Serial0/1
    Router(config-if)#service-policy output MY_POLICY
    Router(config-if)#exit
    Router(config)#

    ! ------------------------------------
    ! NOTES
    ! ------------------------------------

    Router(config)#class-map [match-all|match-any] class_name
  • match-all - the class must match all the succeeding criteria
  • match-any - the class must match any of the succeeding criteria
  • if not specified, defaults to match-all

    Router(config-cmap)#match {protocol|access-group} value
  • protocol - based on known traffic classes via NBAR
  • access-group - based on ACLs
  • not limited to the above criteria; other criteria include class-map (i.e. nested class-maps), CoS, DSCP, IP Precedence, input-interface, MAC address, QoS group, UDP Port Ranges

    Router(config-if)#service-policy {input|output} policy-name
  • only one policy per direction per interface can be applied;
  • that is, each interface can have at most one inbound policy and one outbound policy.

    Other command syntax will be dealt with in another post.
  • Wednesday, April 15, 2009

    IOS: %BGP_MPLS-3-GEN_ERROR


    Mar 18 20:41:38.892 EDT: %BGP_MPLS-3-GEN_ERROR: BGP: MPLS outlabel changed, MPLS forw not updated, prefix not in routing table -Traceback= 10D36950 10D3709C 10B10388 10B10718 10AEEFD0 10AEF030 10B53A50 10B53DC0 10AF588C 10AFD610 10AFE8E0 10A44524 10A3B6D4
    Mar 18 20:41:38.892 EDT: %BGP_MPLS-3-GEN_ERROR: BGP: MPLS outlabel changed, MPLS forw not updated, prefix not in routing table -Traceback= 10D36950 10D3709C 10B10388 10B10718 10AEEFD0 10AEF030 10B53A50 10B53DC0 10AF588C 10AFD610 10AFE8E0 10A44524 10A3B6D4
    Mar 18 20:41:38.892 EDT: %BGP_MPLS-3-GEN_ERROR: BGP: MPLS outlabel changed, MPLS forw not updated, prefix not in routing table -Traceback= 10D36950 10D3709C 10B10388 10B10718 10AEEFD0 10AEF030 10B53A50 10B53DC0 10AF588C 10AFD610 10AFE8E0 10A44524 10A3B6D4


    Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-ENTSERVICES-M), Version
    12.2(50)SG1, RELEASE SOFTWARE (fc2)
    Technical Support:
    http://www.cisco.com/techsupport
    Copyright (c) 1986-2009 by Cisco Systems, Inc.
    Compiled Tue 10-Feb-09 00:17 by prod_rel_team
    Image text-base: 0x10000000, data-base: 0x124FED8C

    ROM: 12.2(44r)SG
    Darkside Revision 0, Jawa Revision 11, Tatooine Revision 140, Forerunner Revision 1.74

    MyRouter uptime is 5 days, 3 hours, 12 minutes
    System returned to ROM by power-on
    System restarted at 19:50:40 EDT Fri Mar 13 2009
    System image file is "bootflash:/cat4500e-entservices-mz.122-50.SG1.bin"

    cisco WS-C4900M (MPC8548) processor (revision 2) with 524288K bytes of memory.
    Processor board ID JAE130628BD
    MPC8548 CPU at 1.33GHz, Cisco Catalyst 4900M
    Last reset from PowerUp
    1 Virtual Ethernet interface
    36 Gigabit Ethernet interfaces
    16 Ten Gigabit Ethernet interfaces
    511K bytes of non-volatile configuration memory.

    Configuration register is 0x2102



    CSCse15707: Trace back seen at bgp_ipv4_mpls_label_change.

    http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCse15707


    First Found-In:

  • 12.2(32.8.7)SRB




  • 12.2(32.8.8)SRA




  • 12.4(12.15)PI7e




  • 12.2(31)SB9




  • 12.2(31.4.5)SB11




  • 12.2(31.4.11)SB12




  • 12.2(28.5.24)SB13




  • Fixed-In:

  • 12.2(32.8.63)SR




  • 12.2(33)SRC




  • 12.4(13.8)PI7c




  • 12.2(33)SRB3




  • 12.2(33.2.18)SRB




  • 12.2(33)SB




  • 12.2(32.8.99a)SR133




  • 12.2(31)SB13




  • 12.2(32.8.1)YCA172.24




  • 12.4(21.14.9)PIC1





  • Symptoms: A router may generate the following error message and a traceback:

    %BGP_MPLS-3-GEN_ERROR: BGP: MPLS outlabel changed, MPLS forw not updated, prefix not in routing table

    Conditions: This symptom is observed on a Cisco router that functions in a VPN carrier supporting carrier topology and that is configured for BGP and IPv4.

    Workaround: This is a cosmetic issue, the traceback is harmless and the functionality of the router is not affected.

    CATOS: %SYS-3-PKTBUFFERFAIL_ERRDIS: Packet buffer failure detected.

    %%SYS-3-PKTBUFFERFAIL_ERRDIS: Packet buffer failure detected. Err-disabling port [dec]/[dec]

    Description:

    This message indicates that the default error-detection packet buffer setting is error disabling the port. Whenever a parity failure is detected on the port, ASIC ports are error disabled. [dec]/[dec] is the module number/port number of the error-disabled port.

    Recommended Action:

    Power cycle the switching module with the error-disabled port. Note The next message appears as four lines.

    Example:
    2009 Mar 19 22:19:18 GMT +00:00 %SYS-3-PKTBUFFERFAIL_ERRDIS:Packet buffer failure detected. Err-disabling port 12/11.
    2009 Mar 19 22:19:19 GMT +00:00 %SYS-3-PKTBUFFERFAIL_ERRDIS:Packet buffer failure detected. Err-disabling port 12/12.




    MySwitch (enable) show port errdisable-timeout 12/12
    Module 12 is not a Komodo+ Firewall
    Module 12 is not a Venus SLB

    Port Status ErrDisable Reason Port ErrDisableTimeout Action on Timeout
    ---- ---------- ------------------- ---------------------- -----------------
    12/12 errdisable packet-buffer-error Enable No Change
    MySwitch (enable)




    MySwitch (enable) show port errdisable-timeout 12/11
    Module 12 is not a Komodo+ Firewall
    Module 12 is not a Venus SLB

    Port Status ErrDisable Reason Port ErrDisableTimeout Action on Timeout
    ---- ---------- ------------------- ---------------------- -----------------
    12/11 errdisable packet-buffer-error Enable No Change
    MySwitch (enable)




    MySwitch (enable) sh port status 12
    Port Name Status Vlan Duplex Speed Type
    ----- -------------------- ---------- ---------- ------ ----- ------------
    12/1 FOLPT1412 connected 562 full 100 10/100BaseTX
    12/2 FOLPT1413 connected 562 full 100 10/100BaseTX
    12/3 3/6 splpw232131 errdisable 562 full 100 10/100BaseTX
    12/4 ldnpsmeg025 errdisable 565 full 100 10/100BaseTX
    12/5 LDNPSM14006 errdisable 565 full 100 10/100BaseTX
    12/6 LDNPSM14007 errdisable 565 full 100 10/100BaseTX
    12/7 LDNPSM14008 errdisable 565 full 100 10/100BaseTX
    12/8 LDNPSM02989 errdisable 565 full 100 10/100BaseTX
    12/9 LDNPSM14015 errdisable 565 full 100 10/100BaseTX
    12/10 LDNPSM14014 errdisable 565 full 100 10/100BaseTX
    12/11 LDNPSM14012 errdisable 565 full 100 10/100BaseTX
    12/12 lsc42n02-app2 errdisable 562 full 100 10/100BaseTX


    MySwitch (enable) show port errdisable-timeout 12/3
    Module 12 is not a Komodo+ Firewall
    Module 12 is not a Venus SLB

    Port Status ErrDisable Reason Port ErrDisableTimeout Action on Timeout
    ---- ---------- ------------------- ---------------------- -----------------
    12/3 errdisable packet-buffer-error Enable No Change
    MySwitch (enable)





    Example:Resolution:

    set module power down 12
    set module power up 12



    Related document:

    Best Practices for Catalyst 4500/4000, 5500/5000, and 6500/6000 Series Switches Running CatOS Configuration and Management

    Thursday, March 19, 2009

    PIX 6.X - Configuring Logical / VLAN interfaces

    Scenario:
    The inside/ethernet1 interface of the PIX will be mapped to two VLANs, VLAN1 with IP address 192.168.1.2/24 and VLAN2 with IP address 192.168.2.2. The outside interface has IP address 10.199.248.225/24

    Topology:



    [Thanks to former colleague Dan for the image.]


    PIX 6 Configuration:

    interface ethernet1 auto
    nameif ethernet1 inside security100
    address inside 192.168.1.2 255.255.255.0

    interface ethernet1 vlan2 logical
    nameif vlan2 inside2 security50
    address vlan2 192.168.2.2 255.255.255.0

    Notes:
    Your Physical Interface is (by default), your VLAN1.


    PIX 7 Configuration:

    interface Ethernet1
    nameif inside
    security-level 100
    ip address 192.168.1.2 255.255.255.0
    no shut

    interface Ethernet1.2
    vlan 2
    nameif inside2
    security-level 50
    ip address 192.168.2.2 255.255.255.0
    no shut