Case Study: D51 BIGIPv9 cannot be accessed via Web User Interface

Device:
Hardware: BIG-IP 5100 (D51)
Software: BIG-IP 9.2.3 Build 34.3

Problem / Symptoms:
- Device access via the GUI results in a page containing only "Configuration Utility is restarting";
- Device access via SSH and console works as expected.

Error Messages Encountered:
- GUI: Only message is "Configuration Utility is restarting" and that's it;
- SSH/Console: repeatedly gets the message "tomcat4 can't start because /var is at 100%"

Root Cause:
Most UNIX (Linux)-based network devices use Tomcat as the webserver, which then enables any web-based service (in this case, the Management GUI) on the device.

In this case, an existing bug in the current OS version causes the system to create jar_cache files which accumulates in the /var/cache/tomcat4/temp directory, which may eventually lead to the Java Virtual Machine (JVM) halting unexpectedly.

As in this case, the 100% of the /var partition has been used up.

Workaround:
1. Login (via SSH or console) into the device. As soon as you login, you should see the tomcat4 failing to startup messages such as the following:

Jun 19 11:04:40 x81glx04 LOGIN: tomcat4 can't start because /var is at 100%.
Jun 19 11:04:50 x81glx04 LOGIN: tomcat4 can't start because /var is at 100%.
Jun 19 11:05:00 x81glx04 LOGIN: tomcat4 can't start because /var is at 100%.

2. Verify the available drive space using df:

[root@MyBigIP:Standby] config # df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda5 373652 294142 60217 84% /
/dev/hda7 57481 7688 46825 15% /config
/dev/hda8 47900 47456 0 100% /var
none 190508 5828 184680 4% /shared/rrd.1.0
none 190508 40 190468 1% /dev/shm
none 190508 1356 189152 1% /var/tmpfs
[root@MyBigIP:Standby] config #

You can check the actual space used up by the /var/cache/tomcat4/temp directory using the du command:

[root@MyBigIP:Standby] config # cd /var/cache/tomcat4
[root@MyBigIP:Standby] tomcat4 # du -h
204K ./temp
1.0K ./work/Standalone/localhost/tmui
2.0K ./work/Standalone/localhost
3.0K ./work/Standalone
4.0K ./work
209K .

3. Stop the tomcat4 daemon: bigstart stop tomcat4

4. Delete out the files in /var/cache/tomcat4/temp directory. Alternatively, using any SFTP application such as WinSCP, copy out the jar_cache files, in case your vendor wants to take a look at them.

5. Restart the tomcat4 daemon: bigstart start tomcat4

6. Verify the disk space available using df:

[root@MyBigIP:Standby] cache # df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda5 373652 295704 58655 84% /
/dev/hda7 57481 7711 46802 15% /config
/dev/hda8 47900 24600 20827 55% /var
none 190508 5828 184680 4% /shared/rrd.1.0
none 190508 40 190468 1% /dev/shm
none 190508 1356 189152 1% /var/tmpfs
[root@MyBigIP:Standby] cache #


Permanent Fix:
BIG-IP version upgrade.

References:
SOL7361: The BIG-IP system can create an excessive number of jar_cache files, filling up the /var directory (requires F5 login)