Today when I started oracle database in alternate node of a cluster in dmesg following error is coming
#dmesg | tail
file: table is full
file: table is full
file: table is full
file: table is full
file: table is full
file: table is full
Verified using sar
# sar -v 2 5
HP-UX unixmemoirs03 B.11.31 U ia64 01/20/12
20:35:52 text-sz ov proc-sz ov inod-sz ov file-sz ov
20:35:54 N/A N/A 1577/28000 0 4256/761302 0 35645/3564877
20:35:56 N/A N/A 1574/28000 0 4252/761302 0 35617/3564854
20:35:58 N/A N/A 1573/28000 0 4250/761302 0 35648/3564885
20:36:00 N/A N/A 1574/28000 0 4251/761302 0 35614/35648 9
20:36:02 N/A N/A 1573/28000 0 4249/761302 0 35646/35648 7
From both above we can see that nfile parameter is completely used.
kctune and kcusage not showing nfile parameter in its output.
When I checked the available documentations for hpux kernel parameters all are saying that nfile parameter in HP-UX 11.31 is deprecated.
After lot of trouble shooting and wasting time on different things a simple
#kcusage nfile
Tunable Value Expression
Changes
nfile 35648 35648 Immed
So somebody set it on the server which in turn created the issues
Once I set it to zero the error was gone.
#kctune nfile=0
http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02266682/c02266682.pdf
"nfile(5) OBSOLETE nfile(5)
(Tunable Kernel Parameters)
NAME
nfile - maximum number of open files (system-wide)
VALUES
Failsafe
0
Default
The default value for nfile is 0 (zero). The value of zero means that the system limit usually enforced
by nfile will be disabled (that is, the number of system-wide open files is limited only by available
memory).
Allowed values
The allowed values for nfile are either 0 (zero) or values between the minimum and maximum
(inclusive). The minimum value is 2048. The maximum value is the maximum 32-bit signed integer
value that can be represented in the system. The value is further constrained in that it must be greater
than or equal to two times the per-process open file hard limit (that is, nfile >= 2*maxfiles_lim).
Specify a positive integer value.
DESCRIPTION
This tunable is now private and deprecated. It should no longer be used. The system open file table was restructured in 11iV3 to remove the architectural limitations on this value, and, therefore, nfile is no longer needed. The recommended way to regulate the maximum number of open files on the system is to set the values of maxfiles_lim and nproc appropriately; the theoretical system maximum can be assumed to be maxfiles_lim * nproc.
The nfile tunable defines the maximum number of slots in the system open file table. This number limits
the cumulative number of open files by all processes in the system. In addition to named files (regular
files, directories, links, device files, and so on), other objects that consume slots in the system open file
table include pipes, FIFOs, sockets, and streams. Be aware that the dup and dup2 system calls consume
entries in the per-process file tables but do not consume new slots in the system open file table.
Who Is Expected to Change This Tunable?
This tunable is not expected to be changed.
Restrictions on Changing
This tunable is dynamic; tuning will take effect immediately on the running system. When the value of
nfile is dynamically tuned it cannot be set to a value lower than the number of current open files in the
running kernel. To allow an unlimited number of system-wide open files, the value should be set to the
default.
When Should the Value of This Tunable Be Raised?
This tunable should only be set to a non-zero value to enforce a limit on the number of open files.
What Are the Side Effects of Raising the Value?
By setting this tunable to a positive, non-zero value, the system wide limit will be enforced. By requiring
the system to enforce a global limit, performance of the open() system call and other related system
calls may be impaired.
When Should the Value of This Tunable Be Lowered?
The value of nfile rarely needs to be lowered. The value may be lowered to limit the number of open
files on a system, and this may reduce memory consumption.
What Are the Side Effects of Lowering the Value?
A lower limit may restrict an application’s ability to allocate a new file descriptor or open a named file.
What Other Tunable Values Should Be Changed at the Same Time?
nfile must be equal to or greater than two times the value of maxfiles_lim. Kernel checks during
tunable setting ensure this.
WARNINGS
This tunable is obsolete and will be removed in a future release of HP-UX.
(Tunable Kernel Parameters)
All HP-UX kernel tunable parameters are release specific. This parameter may be removed or have its
meaning changed in future releases of HP-UX.
Installation of optional kernel software, from HP or other vendors, may cause changes to tunable parameter
values. After installation, some tunable parameters may no longer be at the default or recommended
values. For information about the effects of installation on tunable values, consult the documentation for
the kernel software being installed. For information about optional kernel software that was factory
installed on your system, see HP-UX Release Notes at http://www.hp.com/go/hpux-core-docs."
