Problem Description
On Oracle database 10.2.0.3.0 Data Guard with Broker configuration whenever there is an attempting to restart the standby in read only, the following error occurs:
ERROR: WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
ORA-12170: TNS:Connect timeout occurred
PING[ARC6]: Error 3113 when pinging standby
ARC6: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (1089)
The errors occurred in standby database and primary database hangs!
Cause of the Problem
The above error is caused by LGWR process in Oracle Data Guard having RAC environment and it is in fact Oracle bug. The bug number is 7487408.
The bug fires whenever Data Guard is managed by Broker and standby database is shutdown or opened in read only mode. The Primary still tries to ship/send redo streams to standby in SYNC mode and eventually hangs.
Solution of the Problem
- Download the optach 7487408 from oracle support/metalink in order to solve the bug.
- The above bug is fixed in Oracle patchset 10.2.0.5. So you need to upgrade your oracle database to 10.2.0.5.
Showing posts with label Bug. Show all posts
Showing posts with label Bug. Show all posts
Wednesday, September 29, 2010
Tuesday, September 28, 2010
Index on a partitioned table waits on 'row cache lock' in RAC environment
Problem Description
Creating Index on a partitioned table fails with Oracle error ORA-04021. This is a 2-node Oracle RAC environment and in the table there is large number of partitions and sub-partitions. Index creation waits on 'row cache lock'. The same is true while dropping a tablespace. Drop Tablespace hangs on Row Cache Enqueue.
Problem Investigation
From the trace file we get following information.
It seems this process is waiting for exclusive enqueue lock mode on dc_segments:
Cause of the Problem
Index creation waits on 'row cache lock' in RAC environment occurred due to oracle bug.
Oracle named this bug as 'Bug 6321551' in version 10.2.0.2
'Bug 8417354' in version 10.2.0.4
'Bug 6004916' in version 11.0.0.0
Solution of the Problem
The bug is Oracle internal unpublished bug.
- This bug is fixed in Oracle version 10.2.0.5. So apply Oracle 10.2.0.5 patchset if you are running Oracle version 10g.
- If your Oracle database version is 10.2.0.2 or 10.2.0.3 you can apply one-off patch 6004916.
- Another workaround is to create the index with only one node being started (single instance).
i) Stop all RAC instance but one.
ii) Run the index creation script on one node.
iii) After process is completed startup rest of the instances.
Creating Index on a partitioned table fails with Oracle error ORA-04021. This is a 2-node Oracle RAC environment and in the table there is large number of partitions and sub-partitions. Index creation waits on 'row cache lock'. The same is true while dropping a tablespace. Drop Tablespace hangs on Row Cache Enqueue.
Problem Investigation
From the trace file we get following information.
PROCESS 24:Process 24 is executing sql: 0xacafc75e8 which is:
SO: 0xb9e505908, type: 4, owner: 0xb9f351708, flag: INIT/-/-/0x00
(session) sid: 477 trans: 0xb7ce39798, creator: 0xb9f351708, flag: (41) USR/- BSY/-/-/-/-/-
DID: 0001-0018-00000018, short-term DID: 0001-0018-00000019
txn branch: (nil)
oct: 9, prv: 0, sql: 0xacafc75e8, psql: 0xb0266cc18, user: 55/TA
O/S info: user: oracle, term: pts/1, ospid: 16322, machine: rac2
program: sqlplus@rac2 (TNS V1-V3)
application name: SQL*Plus, hash value=3669949024
waiting for 'row cache lock' blocking sess=0x(nil) seq=4750 wait_time=0 seconds since wait started=18
cache id=2, mode=0, request=5
LIBRARY OBJECT HANDLE: handle=acafc75e8 mtx=0xacafc7718(1) cdp=1
name=CREATE INDEX USER_ACTIVITY_IDX ON PROD.USER_ACTIVITY(IMSI, CALLING_IMSI) TABLESPACE HISTORY_IDX_01 LOCAL PARALLEL 32 UNUSABLE
It seems this process is waiting for exclusive enqueue lock mode on dc_segments:
----------------------------------------From the trace file it is not clear who is holding this row cache enqueue.
SO: 0xafb5989f0, type: 50, owner: 0xb7ce39798, flag: INIT/-/-/0x00
row cache enqueue: count=1 session=0xb9e505908 object=0xb004f3690, request=X
savepoint=0x25a3749
row cache parent object: address=0xb004f3690 cid=2(dc_segments)
hash=2997173a typ=11 transaction=(nil) flags=00000000
own=0xb004f3760[0xb004f3760,0xb004f3760] wat=0xb004f3770[0xafb598a20,0xafb598a20] mode=N
status=-/-/-/-/-/-/-/-/-
request=X release=FALSE flags=2
instance lock id=QC 1cd04d4c 3ab28c03
data=
0000006a 000001da 0039918b 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000
----------------------------------------
Cause of the Problem
Index creation waits on 'row cache lock' in RAC environment occurred due to oracle bug.
Oracle named this bug as 'Bug 6321551' in version 10.2.0.2
'Bug 8417354' in version 10.2.0.4
'Bug 6004916' in version 11.0.0.0
Solution of the Problem
The bug is Oracle internal unpublished bug.
- This bug is fixed in Oracle version 10.2.0.5. So apply Oracle 10.2.0.5 patchset if you are running Oracle version 10g.
- If your Oracle database version is 10.2.0.2 or 10.2.0.3 you can apply one-off patch 6004916.
- Another workaround is to create the index with only one node being started (single instance).
i) Stop all RAC instance but one.
ii) Run the index creation script on one node.
iii) After process is completed startup rest of the instances.
Saturday, June 26, 2010
Dynamic change of CPU_COUNT causes ORA-600 [Ksrpubwait:Deadlock] ORA-2097 Instance crash
Problem Description
Dynamic change of CPU_COUNT parameter causes oracle bug ORA-600 [ksrpubwait:deadlock]. For example, after issuing command "alter system set cpu_count=5;" it fails with following errors.
ORA-600 [ksrpubwait:deadlock], [parameters to cluster db instances - broadcast channel]
ORA-2097: parameter cannot be modified because specified value is invalid
And then instance crashes.
If you look at stack trace you will see,
ksrpubwait ksrpublish kspbcast kspset0 kxfpCPUCountUpdate ksbcpcb
Cause of the Problem
This errors are caused by Oracle Bug 7535429
Solution of the Problem
Solution 01: Bug 7535429 is fixed in version 11.2. So upgrade your oracle database to 11gR2 will solve the problem.
There is one-off patch which also help you to solve the problem. Check for the availability of one-off Patch 7535429 for your platform on MetaLink.
Solution 02: Explicitely setting CPU_COUNT to new value.
Solution 03: If you start the instance from pfile instead of using spfile then it will also solve the problem.
Solution 04: Set explicitly PARALLEL_MAX_SERVERS to the current value instead of relying default value.
Dynamic change of CPU_COUNT parameter causes oracle bug ORA-600 [ksrpubwait:deadlock]. For example, after issuing command "alter system set cpu_count=5;" it fails with following errors.
ORA-600 [ksrpubwait:deadlock], [parameters to cluster db instances - broadcast channel]
ORA-2097: parameter cannot be modified because specified value is invalid
And then instance crashes.
If you look at stack trace you will see,
ksrpubwait ksrpublish kspbcast kspset0 kxfpCPUCountUpdate ksbcpcb
Cause of the Problem
This errors are caused by Oracle Bug 7535429
Solution of the Problem
Solution 01: Bug 7535429 is fixed in version 11.2. So upgrade your oracle database to 11gR2 will solve the problem.
There is one-off patch which also help you to solve the problem. Check for the availability of one-off Patch 7535429 for your platform on MetaLink.
Solution 02: Explicitely setting CPU_COUNT to new value.
Solution 03: If you start the instance from pfile instead of using spfile then it will also solve the problem.
Solution 04: Set explicitly PARALLEL_MAX_SERVERS to the current value instead of relying default value.
While startup, instance terminated after ORA-600 [6006] ORA-600 [6856]
Problem Description
While starting up the instance it fails with following errors:
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [6856], [0], [60], [], [], []
and then SMON abnormally crashes the database instance.
Cause of the Problem
These are oracle bugs and these ORA-600 [6006] and ORA-600 [6856] bug occurred whenever undo segments are trying to rollback a failed transaction and cannot.
For error "ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], []" ,
Oracle is undoing an index leaf key operation. If the key is not found, ORA-00600 [6006] is logged.
For error "ORA-00600: internal error code, arguments: [6856], [0], [60], [], [], []" , SMON is trying to recover a dead transaction. But the undo application runs into an internal error (trying to delete a row that is already deleted).
Solution of the Problem
1) Review the trace files and look for the object(s) involved. If the trace file does not have a SQL statement, search on the following: "block dump header"
2) In the block header there will be a seg/obj = hex value. Convert the hex to dec and this will give you the data_object_id.
3) The alert.log may also show the affected object, for example:
ORACLE Instance ORCL (pid = 8) - Error 600 encountered while recovering transaction (9999, 36) on object 45879.
SQL>select owner, object_name from dba_objects where data_object_id = object#;
This will be the object you need to work with.
To implement the solution:
1. shutdown the oracle instance.
SQL> shut immediate
2. set event - event="10513 trace name context forever, level 2" (this event and setting disables transaction recovery which was initiated by SMON). Open the pfile and add the following line inside pfile.
event="10513 trace name context forever, level 2"
3. startup instance
SQL> startup
4.
- If the object is an index - drop and recreate.
- If the object is a table - drop / export / Create Table as Select (CTAS) to change the object ID
5. shutdown the instance
SQL> shut immediate
6. remove the event
Open the pfile and remove the event line that was added.
7. startup the instance
SQL> startup
8. recreate the affected object(s)
While starting up the instance it fails with following errors:
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [6856], [0], [60], [], [], []
and then SMON abnormally crashes the database instance.
Cause of the Problem
These are oracle bugs and these ORA-600 [6006] and ORA-600 [6856] bug occurred whenever undo segments are trying to rollback a failed transaction and cannot.
For error "ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], []" ,
Oracle is undoing an index leaf key operation. If the key is not found, ORA-00600 [6006] is logged.
For error "ORA-00600: internal error code, arguments: [6856], [0], [60], [], [], []" , SMON is trying to recover a dead transaction. But the undo application runs into an internal error (trying to delete a row that is already deleted).
Solution of the Problem
1) Review the trace files and look for the object(s) involved. If the trace file does not have a SQL statement, search on the following: "block dump header"
2) In the block header there will be a seg/obj = hex value. Convert the hex to dec and this will give you the data_object_id.
3) The alert.log may also show the affected object, for example:
ORACLE Instance ORCL (pid = 8) - Error 600 encountered while recovering transaction (9999, 36) on object 45879.
SQL>select owner, object_name from dba_objects where data_object_id = object#;
This will be the object you need to work with.
To implement the solution:
1. shutdown the oracle instance.
SQL> shut immediate
2. set event - event="10513 trace name context forever, level 2" (this event and setting disables transaction recovery which was initiated by SMON). Open the pfile and add the following line inside pfile.
event="10513 trace name context forever, level 2"
3. startup instance
SQL> startup
4.
- If the object is an index - drop and recreate.
- If the object is a table - drop / export / Create Table as Select (CTAS) to change the object ID
5. shutdown the instance
SQL> shut immediate
6. remove the event
Open the pfile and remove the event line that was added.
7. startup the instance
SQL> startup
8. recreate the affected object(s)
ORA-600 [17147] ORA-600 [Kghlkrem1] ORA-7445 kghlkremf() database crash
During normal database activity, in alert log the following ORA-00600 error occurred and it causes database crashed.
The call stack is as follows:
kghfrmrg kghfre qsmqktcc ktcdso ktcrcm ktdcmt k2lcom k2send xctctl xctcom_with_options opicom
or
k2send xctctl xctcom_with_options opicom opiodr
or
kghalo ktcccadd kwqidracbk kwqidcpmc kwqidafm0 kwqididqx kpoaqdq
or
kghadd_reserved_extent kghget_reserved_extent kghgex kghfnd
Cause of the Problem
The problem occurred due to oracle bug. The bug involves abnormal termination of queries involving bloom filters. This manifests itself as memory corruptions. The memory was freed, reallocated to some other client, and corrupted but slaves were still writing into it.
Solution of the Problem
Solution 01:
This bug is fixed in oracle database version 10.2.0.4. So apply 10.2.0.4 patchset to solve this problem.
Solution 02:
Workaround, you can set the oracle hidden parameter _bloom_filter_enabled to false.
This parameter can be set dynamically:
If you are using spfile then just set as,
SQL> connect / as sysdba
SQL> alter system set "_bloom_filter_enabled"=false scope=both;
If you use pfile to startup your database remove the scope parameter that is use,
SQL> alter system set "_bloom_filter_enabled"=false;
ORA-00600: internal error code, arguments: [17147], [0xC0000003E4CE10A5], [], [], [], [], [], []If you look at the trace file you no longer will see any sql that causes this bug.
ORA-00600: internal error code, arguments: [KGHLKREM1], [0xC0000003E4CE10B0], [], [], [], []
ORA-7445: exception encountered: core dump [kghlkremf()+33] [SIGSEGV]
The call stack is as follows:
kghfrmrg kghfre qsmqktcc ktcdso ktcrcm ktdcmt k2lcom k2send xctctl xctcom_with_options opicom
or
k2send xctctl xctcom_with_options opicom opiodr
or
kghalo ktcccadd kwqidracbk kwqidcpmc kwqidafm0 kwqididqx kpoaqdq
or
kghadd_reserved_extent kghget_reserved_extent kghgex kghfnd
Cause of the Problem
The problem occurred due to oracle bug. The bug involves abnormal termination of queries involving bloom filters. This manifests itself as memory corruptions. The memory was freed, reallocated to some other client, and corrupted but slaves were still writing into it.
Solution of the Problem
Solution 01:
This bug is fixed in oracle database version 10.2.0.4. So apply 10.2.0.4 patchset to solve this problem.
Solution 02:
Workaround, you can set the oracle hidden parameter _bloom_filter_enabled to false.
This parameter can be set dynamically:
If you are using spfile then just set as,
SQL> connect / as sysdba
SQL> alter system set "_bloom_filter_enabled"=false scope=both;
If you use pfile to startup your database remove the scope parameter that is use,
SQL> alter system set "_bloom_filter_enabled"=false;
ORA-00600 [730] [SPACE LEAK] OR ORA-600 [LIBRARYCACHENOTEMPTYONCLOSE] during shutdown
Problem Description
While shutting down oracle database the following error occurred.
In oracle 10g and 11g,
ORA-00600: internal error code, arguments: [730], [4100736], [space leak], [], [], [], [], [], [], [], [], []
If you observe stack trace it will look like,
ksesic2 <- ksmshu <- opistp_real <- opistp <- opiodr <- ttcpip <- opitsk <- opiino <- opiodr <- opidrv <- sou2o <- main <- start
In oracle 10g the error looks like,
ORA-00600: internal error code, arguments: [LibraryCacheNotEmptyOnClose], [], [] ,[], [], [], [], []
If you observe stack trace it will look like, kglshu <- kqlnfy <- kscnfy <- ksmshu <- opistp_real <- opistp <- opiodr <- ttcpip <- opitsk <- opiino <- opiodr <- opidrv <- sou2o <- opimai_real
Cause of the Problem
The error occurred due to oracle bug. Oracle named this bug as BUG 7572335. This happens during database shutdown when child cursors had been previously marked as kept in the shared pool. Note that this is a non-corruptive error.
Solution of the Problem
Solution 01: This bug is solved in Oracle database 11gR2. So upgrade to Oracle database 11.2 or higher is one solution.
Solution 02: If you are using Oracle database 10gR2 then up to patchset version 10.2.0.4 this bug remained and it is solved in version 10.2.0.5. You can upgrade to oracle version 10.2.0.5 by applying the Patchset, Patch 8202632.
Solution 03: If available for your platform and version, you can apply one off patch 7572335. Download and apply the fix, Patch 7572335
While shutting down oracle database the following error occurred.
In oracle 10g and 11g,
ORA-00600: internal error code, arguments: [730], [4100736], [space leak], [], [], [], [], [], [], [], [], []
If you observe stack trace it will look like,
ksesic2 <- ksmshu <- opistp_real <- opistp <- opiodr <- ttcpip <- opitsk <- opiino <- opiodr <- opidrv <- sou2o <- main <- start
In oracle 10g the error looks like,
ORA-00600: internal error code, arguments: [LibraryCacheNotEmptyOnClose], [], [] ,[], [], [], [], []
If you observe stack trace it will look like, kglshu <- kqlnfy <- kscnfy <- ksmshu <- opistp_real <- opistp <- opiodr <- ttcpip <- opitsk <- opiino <- opiodr <- opidrv <- sou2o <- opimai_real
Cause of the Problem
The error occurred due to oracle bug. Oracle named this bug as BUG 7572335. This happens during database shutdown when child cursors had been previously marked as kept in the shared pool. Note that this is a non-corruptive error.
Solution of the Problem
Solution 01: This bug is solved in Oracle database 11gR2. So upgrade to Oracle database 11.2 or higher is one solution.
Solution 02: If you are using Oracle database 10gR2 then up to patchset version 10.2.0.4 this bug remained and it is solved in version 10.2.0.5. You can upgrade to oracle version 10.2.0.5 by applying the Patchset, Patch 8202632.
Solution 03: If available for your platform and version, you can apply one off patch 7572335. Download and apply the fix, Patch 7572335
Friday, June 25, 2010
In RAC flashback transaction query fails with ORA-600[kcbgtcr_13]
Problem Description
In Oracle database 11gR1 RAC environment flashback transaction queries fails with Oracle bug ORA-600[kcbgtcr_13].
For example,
SQL> SELECT UNDO_SQL FROM FLASHBACK_TRANSACTION_QUERY WHERE XID = '08001C00ACD20901';
ORA-00600: internal error code, arguments: [kcbgtcr_13], [], [], [], [], [], [], [], [], [], [], []
If you investigate into trace file you will see error stack as,
kcbgtcr <- ktuq_get_urec <- ktuqup_get_startslot <- ktuqup_init <- ktuqqp_fetch_next_rec <- ktuqtpgv <- krvxInvokeCallbacks <- krvxrfro_FlashReadOne <- krvxrporr_ProcessOneRedoRecord <- krvxread <- krvxgtsp_GetTxnSingleProcess <- krvxgt <- ktuqfcbk <-qerfxFetch <- opifch2 <- kpoal8 <- opiodr <- ttcpip <- opitsk <- opiino <- opiodr <- opidrv <- sou2o <- opimai_real <- main
Cause of the Problem
This is oracle bug and oracle named this bug as bug 7484261.
Solution of the Problem
This issue is fixed in Oracle database 11gR2. So upgrade your database to 11.2g will solve the problem.
If you are still using 11gR1 then download and apply Patch 7484261 from Metalink. Based on your platform and oracle version download from link https://updates.oracle.com/ARULink/PatchDetails/process_form?patch_num=7484261
In Oracle database 11gR1 RAC environment flashback transaction queries fails with Oracle bug ORA-600[kcbgtcr_13].
For example,
SQL> SELECT UNDO_SQL FROM FLASHBACK_TRANSACTION_QUERY WHERE XID = '08001C00ACD20901';
ORA-00600: internal error code, arguments: [kcbgtcr_13], [], [], [], [], [], [], [], [], [], [], []
If you investigate into trace file you will see error stack as,
kcbgtcr <- ktuq_get_urec <- ktuqup_get_startslot <- ktuqup_init <- ktuqqp_fetch_next_rec <- ktuqtpgv <- krvxInvokeCallbacks <- krvxrfro_FlashReadOne <- krvxrporr_ProcessOneRedoRecord <- krvxread <- krvxgtsp_GetTxnSingleProcess <- krvxgt <- ktuqfcbk <-qerfxFetch <- opifch2 <- kpoal8 <- opiodr <- ttcpip <- opitsk <- opiino <- opiodr <- opidrv <- sou2o <- opimai_real <- main
Cause of the Problem
This is oracle bug and oracle named this bug as bug 7484261.
Solution of the Problem
This issue is fixed in Oracle database 11gR2. So upgrade your database to 11.2g will solve the problem.
If you are still using 11gR1 then download and apply Patch 7484261 from Metalink. Based on your platform and oracle version download from link https://updates.oracle.com/ARULink/PatchDetails/process_form?patch_num=7484261
ORA-00600: internal error code, arguments: [4819]
Problem Description
While reviewing database alert log files it was report an ORA-600 [4819] error followed by ORA-00308 and ORA-27037 errors like below.
ORA-00600: internal error code, arguments: [4819], [], [], [], [], [], [], []
ORA-00308: cannot open archived log '/home/oracle/archive/arc_ARJU_001.arc'
ORA-27037: unable to obtain file status
This problem started to happen because recently database has been switched into NOARCHIVELOG mode from ARCHIVELOG mode.
Cause of the Problem
The problem arises because the LOG_ARCHIVE_DEST_n parameter is still set to point to an archive log destination and the default value for the LOG_ARCHIVE_DEST_STATE_n parameter is 'ENABLE'. Based on these two settings database still attempts to open the required archive log file during transaction recovery.
Solution of the Problem
To resolve this problem, mark the destination as deferred and null out the archive destination, That is
1) Login as sysdba.
SQL> connect / as sysdba
2) Set the archive_dest_state_1 to defer.
SQL> alter system set log_archive_dest_state_1 = defer scope=both;
3) Set log_archive_dest_1 to null.
SQL> alter system set log_archive_dest_1 = '' scope=both;
You are done!
While reviewing database alert log files it was report an ORA-600 [4819] error followed by ORA-00308 and ORA-27037 errors like below.
ORA-00600: internal error code, arguments: [4819], [], [], [], [], [], [], []
ORA-00308: cannot open archived log '/home/oracle/archive/arc_ARJU_001.arc'
ORA-27037: unable to obtain file status
This problem started to happen because recently database has been switched into NOARCHIVELOG mode from ARCHIVELOG mode.
Cause of the Problem
The problem arises because the LOG_ARCHIVE_DEST_n parameter is still set to point to an archive log destination and the default value for the LOG_ARCHIVE_DEST_STATE_n parameter is 'ENABLE'. Based on these two settings database still attempts to open the required archive log file during transaction recovery.
Solution of the Problem
To resolve this problem, mark the destination as deferred and null out the archive destination, That is
1) Login as sysdba.
SQL> connect / as sysdba
2) Set the archive_dest_state_1 to defer.
SQL> alter system set log_archive_dest_state_1 = defer scope=both;
3) Set log_archive_dest_1 to null.
SQL> alter system set log_archive_dest_1 = '' scope=both;
You are done!
Wednesday, March 31, 2010
SEVERE:OUI-67073: Apply Session failed: ApplySession::processLocal() failed in system
Problem Description
OPatch apply Patch failed with the following error.
SEVERE:OUI-67073: Apply Session failed: ApplySession::processLocal() failed in system, inventory
From the OPatch logfile I get the following entries,
INFO:Finish backing up system for rollback at Thu Apr 01 12:12:13 EST 2010
INFO:Start the Apply preScript at Thu Apr 01 12:12:13 EST 2010
SEVERE:OUI-67021:SEVERE:OUI-67073: Apply Session failed: ApplySession::processLocal() failed in system, Return Code = 1
INFO:Finish the Apply preScript at Thu Apr 01 12:12:13 EST 2010
INFO:Execution of PRE script failed, with return value = 1
Cause of the Problem
The problem occurred due to oracle bug 6880880. This bug fire if the OPatch version in the $ORACLE_HOME isn't up to date.
Solution of the Problem
Look at the post http://arjudba.blogspot.com/2010/03/severe-oui-67073-applysession-failed.html where the similar error is described.
To solve above bug you have to use the latest released OPatch 10.2, which is available for download from My Oracle Support patch 6880880 by selecting the 10.2.0.0.0 release. After you download that OPatch, install it and then try to apply the patch set again.
Related Documents
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed
OPatch failed with error code 74 - CheckActiveFilesAndExecutables failed
How to Download Patchset or Opatch from metalink
List of Patchset number in metalink
OPatch apply Patch failed with the following error.
SEVERE:OUI-67073: Apply Session failed: ApplySession::processLocal() failed in system, inventory
From the OPatch logfile I get the following entries,
INFO:Finish backing up system for rollback at Thu Apr 01 12:12:13 EST 2010
INFO:Start the Apply preScript at Thu Apr 01 12:12:13 EST 2010
SEVERE:OUI-67021:SEVERE:OUI-67073: Apply Session failed: ApplySession::processLocal() failed in system, Return Code = 1
INFO:Finish the Apply preScript at Thu Apr 01 12:12:13 EST 2010
INFO:Execution of PRE script failed, with return value = 1
Cause of the Problem
The problem occurred due to oracle bug 6880880. This bug fire if the OPatch version in the $ORACLE_HOME isn't up to date.
Solution of the Problem
Look at the post http://arjudba.blogspot.com/2010/03/severe-oui-67073-applysession-failed.html where the similar error is described.
To solve above bug you have to use the latest released OPatch 10.2, which is available for download from My Oracle Support patch 6880880 by selecting the 10.2.0.0.0 release. After you download that OPatch, install it and then try to apply the patch set again.
Related Documents
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed
OPatch failed with error code 74 - CheckActiveFilesAndExecutables failed
How to Download Patchset or Opatch from metalink
List of Patchset number in metalink
OPatch failed with error code 74 - CheckActiveFilesAndExecutables failed
Problem Description
When applying opatch it fails with OPatch failed with error code 74. Further error investigation shows that Prerequisite check "CheckActiveFilesAndExecutables" failed which is displayed below.
Cause of the Problem
The Prerequisite check failed because one/more oracle files are active, which violates the requirement of opatch apply.
If there is a RAC environment, then before applying patch shutdown all instances and listeners in the RAC environment.
If this is not a RAC environment, shut down all instances and listeners associated with the Oracle home that you are updating.
As you see in the error message
"Following executables are active :
/oracledba/app/oracle/product/10.2.0/db/bin/oracle"
the problem happened because in one instance oracle binary is still active.
There is also another possible reason of Prerequisite check fail due to oracle bug.
Solution of the problem
Ensure that all database instances/services, processes, listeners are shutdown currently those are being patched. You can check whether any process running or not by issuing query,
$ ps -ef | grep oracle
$ ps -ef | grep tns
If any process running you may want to kill that using kill -9 command.
Note that you might also need to exit from sql*plus client tool if you are connected as sysdba.
sql> exit;
If you see there is no oracle database process then possibly you might hit oracle Bug 6006980.
Workaround to solve this bug
1. Copy fuser executable from /bin to /sbin as root user.
2. Re-run the opatch command as "oracle" user.
Related Documents
How to Download Patchset or Opatch from metalink
List of Patchset number in metalink
When applying opatch it fails with OPatch failed with error code 74. Further error investigation shows that Prerequisite check "CheckActiveFilesAndExecutables" failed which is displayed below.
oracle:/nas/sysadm/oracle/psu/HP/9119284>opatch apply
Invoking OPatch 10.2.0.4.9
Oracle Interim Patch Installer version 10.2.0.4.9
Copyright (c) 2009, Oracle Corporation. All rights reserved.
Oracle Home : /oracledba/app/oracle/product/10.2.0/db
Central Inventory : /oracledba/app/oracle/oraInventory
from : /var/opt/oracle/oraInst.loc
OPatch version : 10.2.0.4.9
OUI version : 10.2.0.4.0
OUI location : /oracledba/app/oracle/product/10.2.0/db/oui
Log file location : /oracledba/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch2010-03-31_02-17-24AM.log
Patch history file: /oracledba/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch_history.txt
ApplySession applying interim patch '9119284' to OH '/oracledba/app/oracle/product/10.2.0/db'
Interim patch 9119284 is a superset of the patch(es) [ 7609058 7609057 7592346 7375617 7375613 7375611 7197583 715 5252 7155251 7155250 7155249 7155248 ] in the Oracle Home
OPatch will rollback the subset patches and apply the given patch.
Execution of 'sh /nas/sysadm/oracle/psu/HP/9119284/custom/scripts/init -apply 9119284 ':
Return Code = 0
Running prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:
Following executables are active :
/oracledba/app/oracle/product/10.2.0/db/bin/oracle
ApplySession failed during prerequisite checks: Prerequisite check "CheckActiveFilesAndExecutables" failed.
System intact, OPatch will not attempt to restore the system
--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1) OUI-67620:Interim patch 9119284 is a superset of the patch(es) [ 7609058 7609057 7592346 7375617 7375613 737561 1 7197583 7155252 7155251 7155250 7155249 7155248 ] in the Oracle Home
--------------------------------------------------------------------------------
OPatch failed with error code 74
Cause of the Problem
The Prerequisite check failed because one/more oracle files are active, which violates the requirement of opatch apply.
If there is a RAC environment, then before applying patch shutdown all instances and listeners in the RAC environment.
If this is not a RAC environment, shut down all instances and listeners associated with the Oracle home that you are updating.
As you see in the error message
"Following executables are active :
/oracledba/app/oracle/product/10.2.0/db/bin/oracle"
the problem happened because in one instance oracle binary is still active.
There is also another possible reason of Prerequisite check fail due to oracle bug.
Solution of the problem
Ensure that all database instances/services, processes, listeners are shutdown currently those are being patched. You can check whether any process running or not by issuing query,
$ ps -ef | grep oracle
$ ps -ef | grep tns
If any process running you may want to kill that using kill -9 command.
Note that you might also need to exit from sql*plus client tool if you are connected as sysdba.
sql> exit;
If you see there is no oracle database process then possibly you might hit oracle Bug 6006980.
Workaround to solve this bug
1. Copy fuser executable from /bin to /sbin as root user.
2. Re-run the opatch command as "oracle" user.
Related Documents
How to Download Patchset or Opatch from metalink
List of Patchset number in metalink
Saturday, January 30, 2010
RMAN-06900, RMAN-06901, ORA-19921 maximum number of 64 rows exceeded
Problem Description
RMAN backups has been successfully completed but from the backup logs the following error have been generated.
Recovery Manager: Release 11.1.0.7.0 - Production on Fri Jan 29 00:01:15 2010
Copyright (c) 1982, 2007, Oracle. All rights reserved.
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 64 rows exceeded
A variation of the above error is,
Recovery Manager: Release 11.1.0.7.0 - Production on Fri Jan 29 00:01:15 2010
Copyright (c) 1982, 2007, Oracle. All rights reserved.
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 128 rows exceeded
Analysis And Solution of the Problem
As soon as you get error message RMAN-06900 and RMAN-06901 you immediately look for associated error messages. The associated error message should tell you more information and you need to look for those messages in order to solve this error. For example here we are getting additional error "ORA-19921: maximum number of 128 rows exceeded". So our solution will lie on ORA-19921.
RMAN gives a warning message of RMAN-6900, RMAN-6901 ORA-19921 when the output is too huge and oracle is not able to write the log output into v$rman_output. There are several reasons when oracle will not be able to write the log output into V$RMAN_OUTPUT. For example, one of your control file becomes corrupted, hence oracle is unable to write the the log output to RMAN view/table. The another major reasons cause this problem to happen is due to oracle bug.
You get oracle error "ORA-19921: maximum number of 64 rows exceeded" due to oracle Bug 465973.
You get oracle error "ORA-19921: MAXIMUM NUMBER OF 128 ROWS EXCEEDED" due to oracle Bug 8264365.
The bug 465973 is fixed in Oracle 10.2.0.4 and Oracle 11G. So if you see oracle only gives warning message and backup successfully done then you can simply ignore error messages or upgrade oracle or apply patch where bug is fixed.
However if you see your backup is not done due to these errors then immediately check for additional messages. If it happened due to controlfiles then take care of those errors.
Related Documents:
RMAN backups has been successfully completed but from the backup logs the following error have been generated.
Recovery Manager: Release 11.1.0.7.0 - Production on Fri Jan 29 00:01:15 2010
Copyright (c) 1982, 2007, Oracle. All rights reserved.
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 64 rows exceeded
A variation of the above error is,
Recovery Manager: Release 11.1.0.7.0 - Production on Fri Jan 29 00:01:15 2010
Copyright (c) 1982, 2007, Oracle. All rights reserved.
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 128 rows exceeded
Analysis And Solution of the Problem
As soon as you get error message RMAN-06900 and RMAN-06901 you immediately look for associated error messages. The associated error message should tell you more information and you need to look for those messages in order to solve this error. For example here we are getting additional error "ORA-19921: maximum number of 128 rows exceeded". So our solution will lie on ORA-19921.
RMAN gives a warning message of RMAN-6900, RMAN-6901 ORA-19921 when the output is too huge and oracle is not able to write the log output into v$rman_output. There are several reasons when oracle will not be able to write the log output into V$RMAN_OUTPUT. For example, one of your control file becomes corrupted, hence oracle is unable to write the the log output to RMAN view/table. The another major reasons cause this problem to happen is due to oracle bug.
You get oracle error "ORA-19921: maximum number of 64 rows exceeded" due to oracle Bug 465973.
You get oracle error "ORA-19921: MAXIMUM NUMBER OF 128 ROWS EXCEEDED" due to oracle Bug 8264365.
The bug 465973 is fixed in Oracle 10.2.0.4 and Oracle 11G. So if you see oracle only gives warning message and backup successfully done then you can simply ignore error messages or upgrade oracle or apply patch where bug is fixed.
However if you see your backup is not done due to these errors then immediately check for additional messages. If it happened due to controlfiles then take care of those errors.
Related Documents:
How to Restore the Controlfile from Backup.
ORA-00214: Controlfile Version Inconsistent on Startup or Shutdown
Controlfile in Oracle Database.
New Feature of 10.2g: Eliminate Control File Re-Creation
Creating controlfile fails with ORA-01503, ORA-01161
Backup Database control file -User Managed
Recover database after only lose of all controlfiles
Wednesday, December 16, 2009
ORA-00600: internal error code, arguments: [2141]
Problem Description
While starting up or shutting down the database it fails with oracle error message ORA-00600: with the first argument 2141. A sample message from my screen,
SQL> shutdown immediate;
ORA-00600: internal error code, arguments: [2141], [4192191924], [0], [], [], [], [], []
Cause of the Problem
The problem happened due to incorrect control files. If you have different versions of controlfile or if you have wrong controlfile pointed inside spfile/pfile then while startup the database above bug will fire. Similarly after starting up the database if you replace your controlfile by wrong one then while shutting down the database above bug will fire.
Solution of the Problem
The solution of this problem is to correct control_files parameter settings within spfile/pfile whenever you start up the database. Below procedures will help you.
1) Whenever you start up the database make sure whether you start the database by pfile or spfile and also be sure about their location.
2) If you are not sure which file is used then issue "startup nomount" command after connecting as sqlplus / as sysdba to the database.
3) If you start your database with spfile then issuing "show parameter spfile" will locate the location of your spfile location. So simply create a pfile by issuing "create pfile='new_pfile' from spfile" will give you pfile and you can work with that.
4) Whether you start your database with spfile or pfile if you issue "show parameter control_files" it will list the control_files that is denoted by the control_files initialization parameter inside your spfile/pfile (by which one you started your database).
5) Note those locations and check whether the locations are correct. If those are not correct and if you use spfile you can change the locations by,
SQL> alter system set control_files='/oradata1/arju/newcontrol01.ctl', '/oradata1/arju/newcontrol02.ctl' scope=spfile;
and then "startup force". If you use pfile then you need to change manually after opening the pfile with a text editor.
6)If you get error still then check one by one control files as described in the post http://arjudba.blogspot.com/2008/04/ora-00214-controlfile-version.html
If you get this error message while you shutting down your database, then do a shutdown abort and follow the same steps as it is written for startup the database procedures.
Related Documents
http://arjudba.blogspot.com/2009/12/ora-01665-control-file-is-not-standby.html
http://arjudba.blogspot.com/2009/01/ora-00600-arguments-keltnfy-ldminit-46.html
http://arjudba.blogspot.com/2008/12/ora-07445-exception-encountered-core.html
http://arjudba.blogspot.com/2008/11/what-is-ora-00600-internal-error-code.html
http://arjudba.blogspot.com/2008/10/ora-00600-internal-error-code.html
http://arjudba.blogspot.com/2008/09/ora-07445-exception-encountered-core.html
While starting up or shutting down the database it fails with oracle error message ORA-00600: with the first argument 2141. A sample message from my screen,
SQL> shutdown immediate;
ORA-00600: internal error code, arguments: [2141], [4192191924], [0], [], [], [], [], []
Cause of the Problem
The problem happened due to incorrect control files. If you have different versions of controlfile or if you have wrong controlfile pointed inside spfile/pfile then while startup the database above bug will fire. Similarly after starting up the database if you replace your controlfile by wrong one then while shutting down the database above bug will fire.
Solution of the Problem
The solution of this problem is to correct control_files parameter settings within spfile/pfile whenever you start up the database. Below procedures will help you.
1) Whenever you start up the database make sure whether you start the database by pfile or spfile and also be sure about their location.
2) If you are not sure which file is used then issue "startup nomount" command after connecting as sqlplus / as sysdba to the database.
3) If you start your database with spfile then issuing "show parameter spfile" will locate the location of your spfile location. So simply create a pfile by issuing "create pfile='new_pfile' from spfile" will give you pfile and you can work with that.
4) Whether you start your database with spfile or pfile if you issue "show parameter control_files" it will list the control_files that is denoted by the control_files initialization parameter inside your spfile/pfile (by which one you started your database).
5) Note those locations and check whether the locations are correct. If those are not correct and if you use spfile you can change the locations by,
SQL> alter system set control_files='/oradata1/arju/newcontrol01.ctl', '/oradata1/arju/newcontrol02.ctl' scope=spfile;
and then "startup force". If you use pfile then you need to change manually after opening the pfile with a text editor.
6)If you get error still then check one by one control files as described in the post http://arjudba.blogspot.com/2008/04/ora-00214-controlfile-version.html
If you get this error message while you shutting down your database, then do a shutdown abort and follow the same steps as it is written for startup the database procedures.
Related Documents
http://arjudba.blogspot.com/2009/12/ora-01665-control-file-is-not-standby.html
http://arjudba.blogspot.com/2009/01/ora-00600-arguments-keltnfy-ldminit-46.html
http://arjudba.blogspot.com/2008/12/ora-07445-exception-encountered-core.html
http://arjudba.blogspot.com/2008/11/what-is-ora-00600-internal-error-code.html
http://arjudba.blogspot.com/2008/10/ora-00600-internal-error-code.html
http://arjudba.blogspot.com/2008/09/ora-07445-exception-encountered-core.html
Friday, January 30, 2009
ORA-00600 arguments: [keltnfy-ldmInit], [46], [1]
Problem Description
In oracle 10.2.0.1 while creating database with dbca it fails with message below.
ORA-00600: internal error code, arguments: [keltnfy-ldmInit], [46], [1], [], [], [], [], []
If you try to create your database manually then also the command startup nomount fails with above error message.
Cause of the Problem
In the error ldmInit indicates that the problem is related while getting host information of oracle during startup. The first argument 46 indicates the exception LDMERR_HOST_NOT_FOUND which is "gethostbyname system call fails". Oracle was unable to get host information from OS and bug fires.
Solution of the Problem
Step 01: Check permission on /etc/hosts
Note that you need read permission of all users.
Step 02: Check the contents of /etc/hosts
Open the contents of /etc/hosts and check the contents inside it.
$ less /etc/hosts
Note that the contents of this files follow following format.
A simple example,
Step 03: Check the hostname and make sure you can ping your hostname.
$ hostname
database
$ ping database
PING database (127.0.0.1) 56(84) bytes of data.
64 bytes from database (127.0.0.1): icmp_seq=0 ttl=64 time=0.057 ms
64 bytes from database (127.0.0.1): icmp_seq=1 ttl=64 time=0.050 ms
64 bytes from database (127.0.0.1): icmp_seq=2 ttl=64 time=0.041 ms
If you get the following message,
$ ping database
ping: unknown host database
then possibly you will hit above bug.
And you need to modify /etc/hosts files. In the alias section you can give the name of your machine name. If your machine name is "database" you can give /etc/hosts entry as,
And then ping database again. Make sure you are able to ping your host.
Step 04: Diagnosis DNS problem if you have DNS setup
If you have DNS setup, ping is not a tool to diagnose DNS problem. A better tool to use is nslookup, dnsquery, or dig.
$nslookup www.google.com
The forward and reverse lookup should succeed.
Step 05: Check nsswitch.conf
Make sure host lookup is also done through the /etc/hosts file and not just dns. The keyword files should come before dns.
Step 06: Check resolv.conf
Make sure nameserver with DNS name is added there.
In oracle 10.2.0.1 while creating database with dbca it fails with message below.
ORA-00600: internal error code, arguments: [keltnfy-ldmInit], [46], [1], [], [], [], [], []
If you try to create your database manually then also the command startup nomount fails with above error message.
Cause of the Problem
In the error ldmInit indicates that the problem is related while getting host information of oracle during startup. The first argument 46 indicates the exception LDMERR_HOST_NOT_FOUND which is "gethostbyname system call fails". Oracle was unable to get host information from OS and bug fires.
Solution of the Problem
Step 01: Check permission on /etc/hosts
$ ls -l /etc/hosts
-rw-r--r-- 1 root root 153 Nov 24 2007 /etc/hosts
Note that you need read permission of all users.
Step 02: Check the contents of /etc/hosts
Open the contents of /etc/hosts and check the contents inside it.
$ less /etc/hosts
Note that the contents of this files follow following format.
IP Address fully qualified hostname simple or short hostname Alias
A simple example,
$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 database localhost.localdomain localhost
Step 03: Check the hostname and make sure you can ping your hostname.
$ hostname
database
$ ping database
PING database (127.0.0.1) 56(84) bytes of data.
64 bytes from database (127.0.0.1): icmp_seq=0 ttl=64 time=0.057 ms
64 bytes from database (127.0.0.1): icmp_seq=1 ttl=64 time=0.050 ms
64 bytes from database (127.0.0.1): icmp_seq=2 ttl=64 time=0.041 ms
If you get the following message,
$ ping database
ping: unknown host database
then possibly you will hit above bug.
And you need to modify /etc/hosts files. In the alias section you can give the name of your machine name. If your machine name is "database" you can give /etc/hosts entry as,
127.0.0.1 database localhost.localdomain localhost
And then ping database again. Make sure you are able to ping your host.
Step 04: Diagnosis DNS problem if you have DNS setup
If you have DNS setup, ping is not a tool to diagnose DNS problem. A better tool to use is nslookup, dnsquery, or dig.
$nslookup www.google.com
The forward and reverse lookup should succeed.
Step 05: Check nsswitch.conf
$ cat /etc/nsswitch.conf
hosts: files dns
Make sure host lookup is also done through the /etc/hosts file and not just dns. The keyword files should come before dns.
Step 06: Check resolv.conf
$ cat /etc/resolv.conf
nameserver 4.2.2.2
Make sure nameserver with DNS name is added there.
Related Documents
Saturday, December 6, 2008
ORA-07445: exception encountered: core dump kghalp SIGSEGV ORA-10980
Problem Description
I wanted to do online redefinition on a table. Whenever I run start_redef_table procedure then, while creating materialized view it failed with ORA-03114 and ORA-03113.
Below is the scenario.
SQL> exec dbms_redefinition.start_redef_table('M', 'HISTORY', 'HISTORY_INT');
ERROR:
ORA-03114: not connected to ORACLE
BEGIN dbms_redefinition.start_redef_table('M', 'HISTORY', 'HISTORY_INT'); END;
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
In the Alert log file it displays,
Sat Dec 6 16:34:40 2008
Errors in file /var/opt/dumpfile/udump/arjudb_ora_12860.trc:
ORA-07445: exception encountered: core dump [kghalp()+58] [SIGSEGV] [Address not mapped to object] [0x000000068] [] []
ORA-10980: prevent sharing of parsed query during Materialized View query generation
Sat Dec 6 16:36:39 2008
Cause of the Problem
This is an oracle bug. When auditing is enabled and creating materialized view or executing start_redef_table and a Commit/Rollback/Abort transaction is done, memory which is being cleaned up is accessed leading to a dump.
Solution of the Problem
As this problem is an oracle bug and this bug fires when auditing is enabled so there are two solutions.
1)Disable Audit and Restart database:
SQL> alter system set audit_trail=NONE scope=spfile;
System altered.
Since audit_trail is a static parameter so it is needed to bounce the database.
SQL> shutdown immediate;
SQL> startup
2)Apply Patch 10.2.0.3:
This bug is fixed on 10.2.0.3. So apply patch set 10.2.0.3.
I wanted to do online redefinition on a table. Whenever I run start_redef_table procedure then, while creating materialized view it failed with ORA-03114 and ORA-03113.
Below is the scenario.
SQL> exec dbms_redefinition.start_redef_table('M', 'HISTORY', 'HISTORY_INT');
ERROR:
ORA-03114: not connected to ORACLE
BEGIN dbms_redefinition.start_redef_table('M', 'HISTORY', 'HISTORY_INT'); END;
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
In the Alert log file it displays,
Sat Dec 6 16:34:40 2008
Errors in file /var/opt/dumpfile/udump/arjudb_ora_12860.trc:
ORA-07445: exception encountered: core dump [kghalp()+58] [SIGSEGV] [Address not mapped to object] [0x000000068] [] []
ORA-10980: prevent sharing of parsed query during Materialized View query generation
Sat Dec 6 16:36:39 2008
Cause of the Problem
This is an oracle bug. When auditing is enabled and creating materialized view or executing start_redef_table and a Commit/Rollback/Abort transaction is done, memory which is being cleaned up is accessed leading to a dump.
Solution of the Problem
As this problem is an oracle bug and this bug fires when auditing is enabled so there are two solutions.
1)Disable Audit and Restart database:
SQL> alter system set audit_trail=NONE scope=spfile;
System altered.
Since audit_trail is a static parameter so it is needed to bounce the database.
SQL> shutdown immediate;
SQL> startup
2)Apply Patch 10.2.0.3:
This bug is fixed on 10.2.0.3. So apply patch set 10.2.0.3.
Related Documents
Saturday, November 15, 2008
What is an ORA-00600? : internal error code
Generally whenever you got ORA-00600 error on your screen or in the alert log or in trace files they indicate that something inconsistency happened with your oracle database software. This is raised from the kernel code of the Oracle RDBMS software when an unexpected condition is met. This mean exception encountered in the RDBMS kernel code.
A typical form of ORA-00600 error in the alert log is like below,
Errors in file /var/opt/dumpfile/udump/orastdby_ora_15862.trc:
ORA-00600: internal error code, arguments: [15160], [], [], [], [], [], [], []
Sat Nov 15 17:49:27 2008
•Whenever an ORA-600 error is raised a trace file is generated and an entry written to the alert.log with details of the trace file location.
•The trace file contains vital information about what led to the error condition.
•The trace file will be written to either USER_DUMP_DEST (if the error was caught in a user process) or BACKGROUND_DUMP_DEST (if the error was caught in a background process like PMON or SMON). In the above example we see trace file is generated in udump directory which indicates the error was caught in a user process.
•Besides with ORA-00600 we see a list of arguments in square brackets. The first argument tells us from where in the code the error was caught and thus it is the key information in identifying the problem. This argument is either a number or a character string.
•Subsequent arguments are used to supply further information eg. values of internal variables etc.
Related Documents
ORA-00600 internal error code, kmgs_pre_process_request_6
What is Alert Log?
A typical form of ORA-00600 error in the alert log is like below,
Errors in file /var/opt/dumpfile/udump/orastdby_ora_15862.trc:
ORA-00600: internal error code, arguments: [15160], [], [], [], [], [], [], []
Sat Nov 15 17:49:27 2008
•Whenever an ORA-600 error is raised a trace file is generated and an entry written to the alert.log with details of the trace file location.
•The trace file contains vital information about what led to the error condition.
•The trace file will be written to either USER_DUMP_DEST (if the error was caught in a user process) or BACKGROUND_DUMP_DEST (if the error was caught in a background process like PMON or SMON). In the above example we see trace file is generated in udump directory which indicates the error was caught in a user process.
•Besides with ORA-00600 we see a list of arguments in square brackets. The first argument tells us from where in the code the error was caught and thus it is the key information in identifying the problem. This argument is either a number or a character string.
•Subsequent arguments are used to supply further information eg. values of internal variables etc.
Related Documents
ORA-00600 internal error code, kmgs_pre_process_request_6
What is Alert Log?
Sunday, October 19, 2008
ORA-00600 internal error code, kmgs_pre_process_request_6
Error Description
In our production server while doing data pump import operation at the end of import it fails with following errors, then the instance crashed and database went down.
Errors in file /var/opt/dumpfile/bdump/arju_mman_23373.trc:
ORA-00600: internal error code, arguments: [kmgs_pre_process_request_6], [4], [2
], [0], [3], [0xC992C04D0], [], []
Sun Oct 19 18:16:20 2008
MMAN: terminating instance due to error 822
Instance terminated by MMAN, pid = 23373
In the trace file we got the call stack as,
ksedst, ksedmp, ksfdmp, kgerinv, kgeasnmierr, kmgs_pre_process_request, kmgsdrv, ksbabs, ksbrdp, opirip, opidrv, sou2o, opimai_real, main.
Cause of the problem
As we know in Oracle ORA-600 error series are oracle bug. This is oracle Bug 4433838. From alert log we see our MMAN (Memory Manager) process has been terminated. So something wrong happen with oracle MMAN.
This bug actually fires when the initialization parameter SGA_TARGET is set to an exact multiple of 4Gb. So, while setting big SGA be conscious.
A little bit more about this bug is if the SGA_TARGET value in HEXADECIMAL has 00000000 as the last bytes then you can hit this bug.
Detail Explanation of this bug
In out production server we set our SGA_TARGET=48G and this bug fires. We see 48G is multiple of 12 times of 4G. So this bug can error. A little bit elaborate of this issue,
Of 48G we get decimal value as,
SQL> col dec for 9999999999999
SQL> select 48*1024*1024*1024 dec from dual;
DEC
--------------
51539607552
Now let's get hexadecimal value of this number,
SQL> select to_char(51539607552,'XXXXXXXXXXXXX') hex from dual;
HEX
---------------
C00000000
As we see in the hexadecimal representation we get last 8 letters are all 0s; 00000000. The bug happens whenever in hexadecimal representation the 4 bytes have 0. Here in setting of 48G we get last 4 bytes all 0. As for last 0 in binary it represent 0000. So 2 letters of 0 in hexadecimal just form 1 byte(8 bits). So last 8 letters in hexadecimal means we got last 4 bytes 0. And forming this type causes to fire hug.
Solution of the problem
Solution 1)
As bug fires whenever in HEXADECIMAL representation we have 00000000 as the last 4 bytes(8 letters in hex value). So our target is to set SGA_TARGET so that it does not form like this way(4 bytes 0 at last). We can do this if we set our SGA_TARGET not exact multiple of 4G.
Doing set our SGA_TARGET to 47G would happen not the last 4 bytes as all 0s of binary representation.
ALTER SYSTEM SET SGA_TARGET=47G SCOPE=BOTH;
SQL> select to_char(47*1024*1024*1024,'XXXXXXXXXXXXX') hex from dual;
HEX
---------------
BC0000000
We see last 7 letters as 0, and hence the bug will not fire.
Solution 2)
Out database version was 10.2.0.1; base version. And this bug fires. This bug remains in oracle patchset 10.2.0.2 and 10.2.0.3. And is fixed in 10.2.0.4 patchset. So applying patchset 10.2.0.4 would fix the problem.
Solution 3)
Upgrade your database to 11.1.0.6 also will fix the problem.
Solution 4)
Apply the one-off Patch 4433838.
Related Documents
ORA-07445: exception encountered: core dump SIGSEGV
Crash Recovery Fails With ORA-27067
In our production server while doing data pump import operation at the end of import it fails with following errors, then the instance crashed and database went down.
Errors in file /var/opt/dumpfile/bdump/arju_mman_23373.trc:
ORA-00600: internal error code, arguments: [kmgs_pre_process_request_6], [4], [2
], [0], [3], [0xC992C04D0], [], []
Sun Oct 19 18:16:20 2008
MMAN: terminating instance due to error 822
Instance terminated by MMAN, pid = 23373
In the trace file we got the call stack as,
ksedst, ksedmp, ksfdmp, kgerinv, kgeasnmierr, kmgs_pre_process_request, kmgsdrv, ksbabs, ksbrdp, opirip, opidrv, sou2o, opimai_real, main.
Cause of the problem
As we know in Oracle ORA-600 error series are oracle bug. This is oracle Bug 4433838. From alert log we see our MMAN (Memory Manager) process has been terminated. So something wrong happen with oracle MMAN.
This bug actually fires when the initialization parameter SGA_TARGET is set to an exact multiple of 4Gb. So, while setting big SGA be conscious.
A little bit more about this bug is if the SGA_TARGET value in HEXADECIMAL has 00000000 as the last bytes then you can hit this bug.
Detail Explanation of this bug
In out production server we set our SGA_TARGET=48G and this bug fires. We see 48G is multiple of 12 times of 4G. So this bug can error. A little bit elaborate of this issue,
Of 48G we get decimal value as,
SQL> col dec for 9999999999999
SQL> select 48*1024*1024*1024 dec from dual;
DEC
--------------
51539607552
Now let's get hexadecimal value of this number,
SQL> select to_char(51539607552,'XXXXXXXXXXXXX') hex from dual;
HEX
---------------
C00000000
As we see in the hexadecimal representation we get last 8 letters are all 0s; 00000000. The bug happens whenever in hexadecimal representation the 4 bytes have 0. Here in setting of 48G we get last 4 bytes all 0. As for last 0 in binary it represent 0000. So 2 letters of 0 in hexadecimal just form 1 byte(8 bits). So last 8 letters in hexadecimal means we got last 4 bytes 0. And forming this type causes to fire hug.
Solution of the problem
Solution 1)
As bug fires whenever in HEXADECIMAL representation we have 00000000 as the last 4 bytes(8 letters in hex value). So our target is to set SGA_TARGET so that it does not form like this way(4 bytes 0 at last). We can do this if we set our SGA_TARGET not exact multiple of 4G.
Doing set our SGA_TARGET to 47G would happen not the last 4 bytes as all 0s of binary representation.
ALTER SYSTEM SET SGA_TARGET=47G SCOPE=BOTH;
SQL> select to_char(47*1024*1024*1024,'XXXXXXXXXXXXX') hex from dual;
HEX
---------------
BC0000000
We see last 7 letters as 0, and hence the bug will not fire.
Solution 2)
Out database version was 10.2.0.1; base version. And this bug fires. This bug remains in oracle patchset 10.2.0.2 and 10.2.0.3. And is fixed in 10.2.0.4 patchset. So applying patchset 10.2.0.4 would fix the problem.
Solution 3)
Upgrade your database to 11.1.0.6 also will fix the problem.
Solution 4)
Apply the one-off Patch 4433838.
Related Documents
ORA-07445: exception encountered: core dump SIGSEGV
Crash Recovery Fails With ORA-27067
Monday, September 15, 2008
ORA-07445: exception encountered: core dump SIGSEGV
Problem Symptoms
Whenever I check my alert log file I got the text as
Errors in file /var/opt/oracle/admin/udump/orastdby_ora_31795.trc
ORA-07445: exception encountered: core dump [_intel_fast_memcmp()+38] [SIGSEGV] [Address not mapped to object] [0x0] [] []
After checking the trace file I got,
Call Stack Trace shows:
ksedst ksedmp ssexhd intel_fast_memcmp
Recent Changes of The Database
We have changes the CURSOR_SHARING parameter to SIMILAR from EXACT.
Cause of The Problem
This is oracle bug. Bug number 4456646. When the cursor sharing parameter is not set to EXACT this bug may fire. This occur while literal replacement when there are empty string literals in use. This bug happened in oracle version 10.2.0.1. In our environment it was RHL linux 32 bit production server.
Solution of The Problem
Way 1:
As a workaround disable literal replacement. You can do it by,
ALTER SYSTEM SET cursor_sharing='EXACT' SCOPE=both;
if you use spfile.
Way 2:
This bug is fixed in 10.2.0.2 patchset. So, apply 10.2.0.2 patchset in order to fix the prolem.
Way 3:
Apply One-off patch. Download one-off patch 4456646 from metalink and apply if that is avialable for your OS. For Linux x86 and linux x86-64 only available currently.
Whenever I check my alert log file I got the text as
Errors in file /var/opt/oracle/admin/udump/orastdby_ora_31795.trc
ORA-07445: exception encountered: core dump [_intel_fast_memcmp()+38] [SIGSEGV] [Address not mapped to object] [0x0] [] []
After checking the trace file I got,
Call Stack Trace shows:
ksedst ksedmp ssexhd intel_fast_memcmp
Recent Changes of The Database
We have changes the CURSOR_SHARING parameter to SIMILAR from EXACT.
Cause of The Problem
This is oracle bug. Bug number 4456646. When the cursor sharing parameter is not set to EXACT this bug may fire. This occur while literal replacement when there are empty string literals in use. This bug happened in oracle version 10.2.0.1. In our environment it was RHL linux 32 bit production server.
Solution of The Problem
Way 1:
As a workaround disable literal replacement. You can do it by,
ALTER SYSTEM SET cursor_sharing='EXACT' SCOPE=both;
if you use spfile.
Way 2:
This bug is fixed in 10.2.0.2 patchset. So, apply 10.2.0.2 patchset in order to fix the prolem.
Way 3:
Apply One-off patch. Download one-off patch 4456646 from metalink and apply if that is avialable for your OS. For Linux x86 and linux x86-64 only available currently.
Related Documents
Subscribe to:
Posts (Atom)