Most Common Solutions to FRM-41839 and .tmp Files Not Being Deleted

In this Document

Symptoms

Changes

Cause

Solution

References

APPLIES TO:

Oracle Applications Technology Stack - Version 11.5.10.2 to 12.1.3 [Release 11.5.10 to 12.1]
Information in this document applies to any platform.

SYMPTOMS

*** Checked for relevance on 15-MAR-2012 ***

The forms client process f60webmx writes numerous files to the /tmp directory.  These files can entirely fill the /tmp directory thereby stopping further E-Business Suite activity with error messages such as the following:

FRM-41839: Disk I/O error on temporary record buffer file

 

CHANGES

The /tmp directory is used in several different situations:
- When running long running queries 
- Forms or Reports with lots of data 
- When viewing a log file for a Concurrent Request a *.t file (example: OFkp7gMa.t) is generated.

CAUSE

The /tmp directory is full.

SOLUTION

Here are the most common solutions to this issue:

1. Add more disk space to the temporary directory.

2. Review the forms upgrade note to make sure that you have the latest forms bundle patches. This note is frequently updated with new patches:

Note: 125767.1 Upgrading Developer 6i with Oracle Applications 11i
Note: 437878.1 Upgrading Forms and Reports 10g in Oracle Applications Release 12

3. Update the "s_forms_tmpdir" parameter in the $CONTEXT_FILE to specify the temporary directory location where forms server can create and delete temporary files.

Run Auto-Config after making this change.

4. Set the TMPDIR environment variable to a directory with more disk space and restart the forms server. This will redirect most forms temporary files to a more suitable location.

TMPDIR=/bigtmp; export TMPDIR

5. Create a soft link from /var/tmp to a mount point with more disk space.

6. Check timeout settings to make sure that inactive user sessions are being dropped.
FORMS60_TIMEOUT = (APPL_TOP/<SID>.env)
Heartbeat = ($OA_HTML/bin/appsweb.cfg and $OA_HTML/US/appsbase.htm)

Self Service timeout Profile Options:
ICX: Session Timeout
ICX:Limit Time
ICX:Limit connect

s_sesstimeout (11i $CONTEXT_FILE
s_oc4j_sesstimeout (R12 $CONTEXT_FILE)

Reference the following note for more information on these timeout settings:
Note:269884.1 How to fix Timeout issues in 11i

7. For Concurrent Requests, temporary files get written to /tmp when viewing output and log files. Even though the $APPLTMP is set to a directory other than /tmp, the Concurrent Request will still use /tmp unless the $APPLTMP variable is set in the webserver environment.

You would need to add $APPLTMP to the Apache config file if you want it to write to a different directory other than /tmp. If $APPLTMP is not set, /tmp is used by default.

Refer to the following notes for more information on how to add new parameters to a configuration file:
Note 165195.1 Using AutoConfig to Manage System Configurations with Oracle Applications 11i
Note 270519.1 Ext/Pub Customizing an AutoConfig Environment:

Note: Deletion of temporary files created by viewing log files and output files is controlled by system profile 'RRA:Delete Temporary files'.

8. As documented in unpublished Bug:3006192 and the system administrator manual, write a cron job that, every 5 minutes, deletes files that are older than 30 minutes. There are a number of examples of this on the Internet and specific care is required to tune this to a specific server's needs so that only unnecessary files are deleted.

The .TMP files are coming from the f60webmx processes which would normally clean up these files when they are terminated, however, if the script 'adfrmctl.sh stop' is run, it uses kill -9 to terminate all of the currently running forms processes ( the most immediate way of killing a UNIX process ) this has the side effect of killing any f60webmx processes in memory BEFORE they clean up their .TMP files. Also, if users are exiting or killing a long LOV or a Request before it completes the tmp file will not get cleaned up. This means that .TMP files are a normal residue of bouncing the forms server and every day operations, and a cron job should be implemented to routinely clean them up.

9. Please check whether FORMS60_SCROLL_ALL_BUT_ONE is set to TRUE in the instance having this issue.
For servlet mode it should be set in $IAS_ORACLE_HOME/Apache/Jserv/etc/formservlet.ini and for socket mode it should be set in <TWO_TASK>_<HOST>.env (under $APPL_TOP).
Reference: Usage of FORMS60_SCROLL_ALL_BUT_ONE environment variable Note: 272419.1

10. Sometimes there are site dependent requirements, but generally speaking, temporary spaces are not NFS mounted because of its transient nature. If you suspect that the problem is caused by the NFS mount, simply unmount that space and link it to a spare directory on a local disk.  
Please note, applPtmp location (note "P") on database node must be shared between nodes, if this is a RAC system. 
Reference:
How To Get Forms To Write To A Different /tmp? (Doc ID 1502924.1)
Large Export Temp Files Filling Up /tmp (Doc ID 437197.1)
Why Does Oracle Forms Create .TMP Files Which Fill Up The Filesystem e.g. /tmp ? Note: 162232.1

eg: TMPDIR=/<new path to tmpdir>; export TMPDIR

11. If using a NFS mount, verify if “nolock” is set - reference: http://www-01.ibm.com/support/docview.wss?uid=swg21497633

12. Obtain truss/tusc on the f60webmx process.  Note 110888.1
NOTE: R12 uses frmweb instead of f60webmx 

Note: 748314.1  How To Find The Number of Forms Processes in R12?

Example:  strace -/tmp/truss1.log -afeT -p <PID>
          lsof -p <PID> 2>&1 > /tmp/fnd_lsof.txt

13.  Development has increased tmp limit to 512Gb.
Apply latest forms bundle patch per Note: 437878.1
Then in ORA_CONFIG_HOME/10.1.2/forms/server/default.env
add after:
FORMS_SCROLL_ALL_BUT_ONE
add:
FORMS_RECMGR_RECSIZE_POWER=4

NOTE:
1 = 4Gb
2 = 8Gb
3 = 16Gb
4 = 32Gb
5 = 64Gb
6 = 128Gb
7 = 256Gb
8 = 512Gb

Save
Bounce Forms 
Retest.

If this fixes the issue, then, if using JDK 1.7 apply Patch 14635756 (Apply Patch 13579837 if using JDK 1.6) which delivers a new context variable: FORMS_RECMGR_RECSIZE_POWER
The default FORMS_RECMGR_RECSIZE_POWER=1 which is 4Gb.
Then update FORMS_RECMGR_RECSIZE_POWER in the $CONTEXT_FILE
Run Autoconfig so that the changes are preserved when you run autoconfig.

Need to update the $CONTEXT_FILE, update the parameter then run autoconfig to increase the temp space.

NOTE: The new 4G-512Gb Limit is only for R12 .
11i still has a limit of 2G and there will not be a backport for this fix.

14. Review the notes below if additional information is needed.

REFERENCES

NOTE:437197.1  - Large Export Temp Files Filling Up /tmp
NOTE:110888.1  - How to Trace Unix System Calls
NOTE:437878.1  - Upgrading OracleAS 10g Forms and Reports in Oracle E-Business Suite Release 12
NOTE:748314.1  - How To Find The Number of Forms Processes in R11i & R12 ?
NOTE:272419.1  - Usage of FORMS60_SCROLL_ALL_BUT_ONE environment variable
NOTE:433402.1  - R11i: How to redirect to another file system the heap dump and javacore files which now are dumped to /tmp directory
NOTE:165195.1  - Using AutoConfig to Manage System Configurations with Oracle Applications 11i
NOTE:208591.1  - How to Change the Directories where Applications Temporary Files are Generated
NOTE:270519.1  - Customizing an AutoConfig Environment
NOTE:395767.1  - Where the Forms Server Temporary Files Are Kept on the Disk?
NOTE:431392.1  - javacore and heapdump files are filling up /tmp directory on AIX platform
NOTE:469553.1  - How To Reduce The Size Of Temp File During The Export Process?


NOTE:1165208.1  - Questions on Purge Obsolete Generic File Manager Data
NOTE:125767.1  - Upgrading Developer 6i with Oracle Applications 11i
NOTE:145487.1  - Temporary Files Types .t, .temp and .tmp are Saving Under /var/tmp
NOTE:162232.1  - Why Does Oracle Forms Create .TMP Files Which Fill Up The Filesystem e.g. /tmp ?
NOTE:269884.1  - 11i: How To Fix The Forms Timeout Issue In Oracle Applications
NOTE:969646.1  - 11i and R12 FRM-41839 ERROR WHEN TRYING TO VIEW THE LAST RECORD