Monday, July 5, 2010

FortiOS v3.00 MR5 - CPU Usage Too High

Problem:

Fortigate 3600 running version 3.00 MR5 Patch 2 keeps sending high CPU trap SNMP traps to the SNMP trap servers. CPU utilization is confirmed to be high, based from the output of “get system performance status” or from the GUI. From “diag sys top”, confirmed that the “merged_daemons” process is using 99% of the total CPU, then shortly goes down to 14%.


Cause:
This is due to bug documented below:

0062617: race condition in flgd can cause merged_daemons to spin
The merged_daemons was constantly in the 'R' state and consuming 99% of CPU (when top is first started, the usage will display as 99% -- the usage will decrease to 14% while top is running).

Fix: Build: 0566


Workaround:
Restart merged_daemons as follows:
  • Enter diag sys top and take note of the PID of merged_daemons
  • Enter diagnose sys kill 11 [pid]
Note that merged_daemons may still climb back up to 99%.


Resolution/Workaround:
Upgrade to FortiOS MR6 or later.