FortiOS v3.00 MR5 - CPU Usage Too High

Problem:

Fortigate 3600 running version 3.00 MR5 Patch 2 keeps sending high CPU trap SNMP traps to the SNMP trap servers. CPU utilization is confirmed to be high, based from the output of “get system performance status” or from the GUI. From “diag sys top”, confirmed that the “merged_daemons” process is using 99% of the total CPU, then shortly goes down to 14%.

Cause:
This is due to bug documented below:

---

0062617: race condition in flgd can cause merged_daemons to spin
The merged_daemons was constantly in the 'R' state and consuming 99% of CPU (when top is first started, the usage will display as 99% -- the usage will decrease to 14% while top is running).

Fix: Build: 0566

---

Workaround:
Restart merged_daemons as follows:

• Enter diag sys top and take note of the PID of merged_daemons
• Enter diagnose sys kill 11 [pid]

Note that merged_daemons may still climb back up to 99%.

Resolution/Workaround:
Upgrade to FortiOS MR6 or later.

IOS: %EARL_L3_ASIC-SP-3-INTR_WARN: EARL L3 ASIC: Non-fatal interrupt Packet Parser block interrupt

Dec 18 09:54:43.989 JST: %EARL_L3_ASIC-SP-STDBY-3-INTR_WARN: EARL L3 ASIC: Non-fatal interrupt Packet Parser block interrupt
Dec 18 09:54:43.993 JST: %EARL_L3_ASIC-SP-3-INTR_WARN: EARL L3 ASIC: Non-fatal interrupt Packet Parser block interrupt

Description
These messages are indicating that the switch has received an invalid packet which contained a Layer 3 IP checksum error. These packets are normally being dropped silently within older IOS. In some IOS releases, the switch informs of this condition to warn users that there is (are) devices outside sending IP packets with checksum errors and/or with wrong length.

See CSCdz10360 (Need a CLI to be able to disable L3 error checking in HW) regarding this enhancement.

Workaround
These messages are purely informational. You may either:

1. SPAN all the Vlans and look at layer3 IP source address then remove the device generating invalid packets (unfortunately the switch doesn't track the IP address. The only way is to sniff every suspected Vlan to find out where those invalid packets are coming from).



2. Configure (this is a new config option added by means of CSCdz10360):
   no mls verify ip checksum ---> to stop to check for packet checksum errors
   no mls verify ip length ---> to stop to check for packet length errors
   no mls verify ip length minimum ---> to eliminate check for IP packets that are minimum length.
   no mls verify ip same-address ---> to stop checking for packet having equal source and destination IP address.



3. Do nothing as these are pure informational.

IOS: %ETHCNTR-3-LOOP_BACK_DETECTED : Keepalive packet loop-back detected on [chars]

Scenario
The switch reports this error message, and the port is forced to linkdown:
%ETHCNTR-3-LOOP_BACK_DETECTED : Keepalive packet loop-back detected on [chars]

Oct 2 10:40:13: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected on GigabitEthernet0/1
Oct 2 10:40:13: %PM-4-ERR_DISABLE: loopback error detected on Gi0/1, putting Gi0/1 in err-disable state

Description
The problem occurs because the keepalive packet is looped back to the port that sent the keepalive. Keepalives are sent on the Catalyst switches in order to prevent loops in the network. Keepalives are enabled by default on all interfaces. You see this problem on the device that detects and breaks the loop, but not on the device that causes the loop.

Workaround
Issue the no keepalive interface command in order to disable keepalives. A disablement of the keepalive prevents errdisablement of the interface, but it does not remove the loop.

Permanent Fix
In Cisco IOS Software Release 12.2(x)SE-based releases and later, keepalives are not sent on fiber and uplink interfaces by default. Upgrading the IOS version to this or later images should prevent the above issue in the first place.

Fortigate: Password recovery

To reset the FortiGate unit password:

1. Connect the terminal to the FortiGate unit using the null modem cable.
   
2. Log on at the console with the user name "maintainer" and password "bcpb" followed immediately by the unit serial number. You must enter the alphabetic characters of the serial number in upper case.
   
3. Enter the following commands:
   

   config system admin
   edit admin
   set password mypassword
   end

Special Notes:

• You must first power down the FortiGate unit, and power it up again.
• Follow the above steps within one minute of the restart

If the maintainer login is initially unsuccessful, try the following two tips to ensure successful login:

• You many not have the correct serial #. Copy the serial # displayed on the console during initial boot-up and paste it into a terminal editor window.
• In the terminal editor window, finish composing the full password by adding "bcpb" before the serial # and then copy & paste the entire password into the console.