Too many open files

To determine if the number of open files is growing over a period of time, issue lsof to report the open files against a PID on a periodic basis.  For example:

# lsof -p (PID of process) -r (interval) > lsof.out

Note: The interval is in seconds, 1800 for 30 minutes.

This output does not give the actual file names to which the handles are open. It provides only the name of the file system (directory) in which they are contained. The lsof command indicates if the open file is associated with an open socket or a file. When it references a file, it identifies the file system and the inode, not the file name.

Run the following command to determine the file name:

# df -kP filesystem_from_lsof | awk '{print $6}' | tail -1

Now note the filesystem name. And then run:

# find filesystem_name -inum inode_from_lsof -print

This will show the actual file name.

To increase the number, change or add the nofiles=XXXXX parameter in the /etc/security/limitsfile, run:

# chuser nofiles=XXXXX user_id

You can also use svmon:

# svmon -P java_pid -m | grep pers

This lists opens files in the format: filesystem_device:inode. Use the same procedure as above for finding the actual file name.