Showing posts with label RMAN. Show all posts
Showing posts with label RMAN. Show all posts

Saturday, July 31, 2010

RMAN duplicate fails with RMAN-10006, ORA-03113, ORA-01092

Problem Description
While duplicating database using RMAN it fails with ORA-01092: "ORACLE instance terminated. Disconnection forced" and ora-03113 error reported.
RMAN-00601: fatal error in recovery manager
RMAN-03004: fatal error during execution of command
RMAN-10039: error encountered while polling for RPC completion on channel clone_default
RMAN-10006: error running SQL statement: select act from x$ksusex where sid=:1 and serial=:2
RMAN-10002: ORACLE error: ORA-03113: end-of-file on communication channel
RMAN-03002: failure of Duplicate Db command at 02/30/2010 01:21:18
RMAN-03015: error occurred in stored script Memory Script
RMAN-06136: ORACLE error from auxiliary database:
ORA-01092: ORACLE instance terminated. Disconnection forced
If we look for the auxiliary database alert.log file then it shows that Undo Tablespace 'UNDOTBS1' does not exist or of wrong type.

Cause of the Problem
The UNDO_TABLESPACE parameter of auxiliary database is not same as target database. In the source database undo tablespace is set to UNDOTBS2 while in target database undo tablespace is set to UNDOTBS1. The undo tablespace specified in the auxiliary database parameter file does not exist.

Solution of the Problem
Make sure that the undo tablespace for auxiliary database is same as of target database.
Edit the init.ora or alter the UNDO_TABLESPACE parameter of the auxiliary instance and set
undo_tablespace = UNDOTBS2
- restart the auxiliary instance in nomount mode and repeat the duplicate database command.

RMAN duplicate database fails with RMAN-00601, RMAN-10006, ORA-00704, ORA-39700

Problem Description
Duplicating database using RMAN fails with following errors.
channel channel1: starting piece 1 at 29-JUL-2010 03:49:19
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00601: fatal error in recovery manager
RMAN-03004: fatal error during execution of command
RMAN-10039: error encountered while polling for RPC completion on channel channel1
RMAN-10006: error running SQL statement: select action from gv$session where sid=:1 and serial#=:2 and inst_id=:3
RMAN-10002: ORACLE error: ORA-3114: not connected to ORACLE
RMAN-10041: Could not re-create polling channel context following failure.
RMAN-10024: error setting up for rpc polling
RMAN-10005: error opening cursor
RMAN-10002: ORACLE error: ORA-3114: not connected to ORACLE
RMAN-03002: failure of Duplicate Db command at 02/23/2007 11:11:50
RMAN-03015: error occurred in stored script Memory Script
RMAN-06136: ORACLE error from auxiliary database: ORA-1092: ORACLE instance
terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option

Cause of the Problem
The errors occurred because the Oracle versions for both the target database home and the auxiliary database home are not same.

Solution of the Problem
The solution is to upgrade the auxiliary database server binaries to match the target database server. Also if there is one off patches that exist on the target database, you also need to apply on the auxiliary database in order to make the duplication work.

Thursday, June 10, 2010

crosscheck archivelog fails with validation failed for archived log

Problem Description
Whenever I run "crosscheck archivelog all" it failed with message "validation failed for archived log" like below.
RMAN> crosscheck archivelog all;

released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=323 devtype=DISK
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_10/o1_mf_1_3_5bkc1o5q_.arc recid=6 stamp=697208725
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_10/o1_mf_1_4_5bkd60x4_.arc recid=7 stamp=697209883
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_10/o1_mf_1_5_5bkd78q9_.arc recid=8 stamp=697209922
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_10/o1_mf_1_6_5bkf2rwx_.arc recid=9 stamp=697210802
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_10/o1_mf_1_7_5blbdsqs_.arc recid=10 stamp=697240820
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_10/o1_mf_1_8_5blc2tvx_.arc recid=11 stamp=697241524
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_11/o1_mf_1_9_5bm1t7hk_.arc recid=12 stamp=697264802
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_11/o1_mf_1_10_5bmqwzr6_.arc recid=13 stamp=697287417
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_11/o1_mf_1_11_5bnjqg4n_.arc recid=14 stamp=697312840
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_11/o1_mf_1_12_5bnytoy5_.arc recid=15 stamp=697327280
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_12/o1_mf_1_13_5boplo7l_.arc recid=16 stamp=697351600
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_12/o1_mf_1_14_5bp6vv6w_.arc recid=17 stamp=697368277
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_12/o1_mf_1_15_5bpdtlfn_.arc recid=18 stamp=697374380
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_13/o1_mf_1_16_5br3nh19_.arc recid=19 stamp=697430505
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_13/o1_mf_1_17_5brvlghg_.arc recid=20 stamp=697455015
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_14/o1_mf_1_18_5btzdcv5_.arc recid=21 stamp=697524454
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_14/o1_mf_1_19_5bw0drf0_.arc recid=22 stamp=697558259
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_14/o1_mf_1_20_5bwvyksr_.arc recid=23 stamp=697586476
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_15/o1_mf_1_21_5bxmsqql_.arc recid=24 stamp=697610898
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_15/o1_mf_1_22_5bz0h5on_.arc recid=25 stamp=697656639
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_15/o1_mf_1_23_5bzjbvg2_.arc recid=26 stamp=697672884
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_16/o1_mf_1_24_5c0lgk9r_.arc recid=27 stamp=697707819
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_16/o1_mf_1_25_5c24o21l_.arc recid=28 stamp=697759227
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_17/o1_mf_1_26_5c30s9py_.arc recid=29 stamp=697788035
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_17/o1_mf_1_27_5c4s0wyy_.arc recid=30 stamp=697845623
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_18/o1_mf_1_28_5c5k0jrj_.arc recid=31 stamp=697870186
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_18/o1_mf_1_29_5c6z1g0p_.arc recid=32 stamp=697917319
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_18/o1_mf_1_30_5c7mqfl8_.arc recid=33 stamp=697938502
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_19/o1_mf_1_31_5c89jkhw_.arc recid=34 stamp=697960811
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_19/o1_mf_1_32_5c9ggk5p_.arc recid=35 stamp=697998634
validation succeeded for archived log
archive log filename=/u01/app/oracle/archivelog/2009_12_02/o1_mf_1_343_5kdfc1sv_.arc recid=36 stamp=704560058
validation succeeded for archived log
archive log filename=/u01/app/oracle/archivelog/2009_12_02/o1_mf_1_344_5kdfy3pd_.arc recid=37 stamp=704560667
validation succeeded for archived log
archive log filename=/u05/archive/%t_%s_%r.arc1_345_697204983.dbf recid=38 stamp=704560901
validation succeeded for archived log
archive log filename=/u05/archive/1_346_697204983.dbf recid=39 stamp=704561258
validation succeeded for archived log
archive log filename=/u05/archive/1_347_697204983.dbf recid=40 stamp=704561835
validation succeeded for archived log
archive log filename=/u05/archive/1_348_697204983.dbf recid=41 stamp=704561992
validation succeeded for archived log
archive log filename=/u05/archive/1_349_697204983.dbf recid=42 stamp=704562987
validation succeeded for archived log
archive log filename=/u05/archive/1_350_697204983.dbf recid=43 stamp=704562987
validation succeeded for archived log
archive log filename=/u05/archive/1_351_697204983.dbf recid=44 stamp=704562992
validation succeeded for archived log
archive log filename=/u05/archive/1_352_697204983.dbf recid=45 stamp=704563257
validation succeeded for archived log
archive log filename=/u05/archive/1_353_697204983.dbf recid=46 stamp=704563321
validation succeeded for archived log
archive log filename=/u05/archive/1_354_697204983.dbf recid=47 stamp=704563408
validation succeeded for archived log
archive log filename=/u05/archive/1_355_697204983.dbf recid=48 stamp=704563434
validation succeeded for archived log
archive log filename=/u05/archive/1_356_697204983.dbf recid=49 stamp=704584836
validation succeeded for archived log
archive log filename=/u05/archive/1_357_697204983.dbf recid=50 stamp=704586204
validation succeeded for archived log
archive log filename=/u05/archive/1_358_697204983.dbf recid=51 stamp=704611511
validation succeeded for archived log
archive log filename=/u05/archive/1_359_697204983.dbf recid=52 stamp=704648675
validation succeeded for archived log
archive log filename=/u05/archive/1_360_697204983.dbf recid=53 stamp=704650176
validation succeeded for archived log
archive log filename=/u05/archive/1_361_697204983.dbf recid=54 stamp=704650675
validation succeeded for archived log
archive log filename=/u05/archive/1_362_697204983.dbf recid=55 stamp=704671244
validation succeeded for archived log
archive log filename=/u05/archive/1_363_697204983.dbf recid=56 stamp=704692487
validation succeeded for archived log
archive log filename=/u05/archive/1_364_697204983.dbf recid=57 stamp=704708839
validation succeeded for archived log
archive log filename=/u05/archive/1_365_697204983.dbf recid=59 stamp=704736263
validation succeeded for archived log
archive log filename=/u05/archive/1_366_697204983.dbf recid=58 stamp=704736262
validation succeeded for archived log
archive log filename=/u05/archive/1_367_697204983.dbf recid=60 stamp=704736266
validation succeeded for archived log
archive log filename=/u05/archive/1_368_697204983.dbf recid=61 stamp=704737398
Crosschecked 56 objects
Cause of the Problem
The problem happened because archive log file destination was changed or someone deleted the archive log files manually using OS commands.

Solution of the Problem
The solution is to delete the expired archive log files. So do following,
$ rman
RMAN> connect target /
RMAN> crosscheck archivelog all;
RMAN> delete expired archivelog all;

Tuesday, March 2, 2010

About RMAN Compatibility Matrix

About RMAN Environment
If you say about RMAN Environment, then there comes the following components.
1) RMAN executable
2) Recovery catalog database
3) Recovery catalog schema in the recovery catalog database
4) Target database
5) Auxiliary database (used for duplicate, duplicate for standby or tablespace point-in-time recovery)

Each component has a release number. Oracle uses up to 5 digits to indicate a version of the release, but only up to 4 are significant for RMAN compatibility purposes. For example, to check the release number of RMAN executable just issue, rman on command prompt like below.
E:\Documents and Settings\Arju>rman

Recovery Manager: Release 10.2.0.1.0 - Production on Tue Mar 2 16:16:59 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.


Here we see the version is 10.2.0.1.0

Determine the Recovery Catalog Schema Version

Use SQL*Plus to connect to the recovery catalog database as the catalog owner. For example, enter:

$ sqlplus rman@catdb
Query the rcver catalog table. For example, run this query:

SQL> SELECT * FROM rcver;

VERSION
------------
09.00.01.00
10.02.01.00
11.01.00.03
Note that, if multiple versions are listed, then the last row is the current version, and the rows before it are prior versions. In the preceding example, the current recovery catalog schema version is 11.1 and the previous version was 10.2.

Note that for releases 10.2 and later, the last two digits in the rcver output indicate patch level. For earlier releases, they are always zeros.

RMAN Compatibility Matrix
In general, the rules of RMAN compatibility are as follows:

1) You can create an 8.x or 9.x recovery catalog schema in any Oracle database release 8.1.x (or higher), and a 10.0.1 (or higher) recovery catalog schema in any Oracle database release 9.0.1 (or higher).

2) The version of an auxiliary database instance must be equal to the version of the RMAN client.

3) Any release of Oracle database can restore backup sets and copies created by any prior release back to Oracle8i.

4) The RMAN executable version should be the same as the target database. There are some exceptions regarding this rule.

5) The RMAN catalog schema version must be greater than or equal to the RMAN executable.

6) The RMAN catalog is backwards compatible with target databases from earlier releases.

7) While backing up an Oracle Database 10g or later database with the Oracle9i RMAN client, you cannot include a control file that was created using COMPATIBLE=10.0.0 in a datafile backup set. The workaround is to turn control file autobackup ON.

RMAN Compatibility table is shown below.








Target/Auxiliary
Database
RMAN Executable Catalog Database Catalog Schema
8.0.68.0.6>=8.1.7>=8.0.6
8.1.78.0.6.1>=8.1.7>=8.1.7
8.1.78.1.7>=8.1.7>=RMAN executable
8.1.7.4 8.1.7.4 >=8.1.7 8.1.7.4
8.1.7.4 8.1.7.4 >=8.1.7 >=9.0.1.4
9.0.1 9.0.1>=8.1.7 >= RMAN executable
9.2.0>=9.0.1.3 and <= Target database>=8.1.7>= RMAN executable
10.1.0>=9.0.1.3 and <= Target database>=9.0.1>= RMAN executable
10.2.0>=9.0.1.3 and <= target database executable>=9.0.1>= RMAN executable
11.1.0>=9.0.1.3 and <= target database executable>=9.0.1>= RMAN executable
11.2.0>=9.0.1.3 and <= target database executable>=9.0.1>= RMAN executable


From above table you can easily conclude about the compatibility about different components of RMAN.

Note that when compatibility matrix is broken the following error message may be generated.
- RMAN-06186: PL/SQL package string.string version string in string database is too old
- RMAN-06429: string database is not compatible with this version of RMAN

Related Documents
RMAN-06091 no channel allocated for maintenance (of an appropriate type)
RMAN-06429: TARGET database is not compatible with this version of RMAN
How to take RMAN backup on a remote disk/ location
RMAN-00554,RMAN-04005,ORA-0103 when remote connection by rman
Set Date format inside RMAN environment
How to skip a tablespace for restore operation
How to debug Backup, Restore Session in RMAN
Restore operation fails with RMAN-11003 ORA-01511 ORA-01516
Recover database after only lose of all controlfiles
RMAN-04014: startup failed: ORA-07446: sdnfy: bad value
How to restore an Spfile from autobackup older than 7 days
RMAN-06172: no autobackup found
Creating a Duplicate Database on a Remote Host -Part1
Database Duplication Fails Missing Log RMAN-06053 RMAN-06025
How to perform Database Point in time Recovery DBPITR

Wednesday, February 17, 2010

RMAN-06900 RMAN-06901 ORA-19921: maximum number of 64 rows exceeded

Problem Description
RMAN backup completed successfully but backup logs shows warning message as,
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows ORACLE error from target database:
ORA-19921: maximum number of 64 rows exceeded

A variant version of warning message is like below,

connected to target database: PROD1 (DBID=3378321056)
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 128 rows exceeded

Cause of the Problem
The RMAN-06900 RMAN-06901 ORA-19921 error stack indicates oracle RMAN bug. ORA-19921: maximum number of 64 rows exceeded is oracle bug 4659734 and ORA-19921: MAXIMUM NUMBER OF 128 ROWS EXCEEDED is oracle Bug 8264365. This bug fires when the RMAN output is too huge because it couldn't log the output in v$rman_output.

Solution of the Problem
If your RMAN goes well and you only see warning message in your RMAN log then simply you can ignore these warnings. The bug "ORA-19921: maximum number of 64 rows exceeded" is fixed in Oracle 10.2.0.4 version and in 11g. So to solve it upgrade your oracle version or apply oracle patch. If you already have oracle version 10.2.0.4 + and hit this bug then in fact you should do nothing and just to wait for oracle to resolve the issue.

Related Documents
http://arjudba.blogspot.com/2010/01/rman-00571-rman-00569-rman-00571-rman.html
http://arjudba.blogspot.com/2010/01/rman-06900-rman-06901-ora-19921-maximum.html

ORA-00204 ORA-00202 ORA-27071 HP-UX Error: 9: Bad file number

Problem Description
RMAN backup fails with ORA-00204 ORA-00202 ORA-27071 HP-UX Error: 9: Bad file number as below.

Recovery Manager: Release 11.1.0.7.0 - Production on Fri Jan 29 00:01:15 2010
Copyright (c) 1982, 2007, Oracle. All rights reserved.
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-00204: error in reading (block 1, # blocks 1) of control file
ORA-00202: control file: '/SID/oradata/APEXP/control01.ctl'
ORA-27071: unable to seek to desired position in file
HP-UX Error: 9: Bad file number

An another version of error message appear while starting RMAN is,
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00554: initialization of internal recovery manager package failed
RMAN-06003: ORACLE error from target database:
ORA-00204: error in reading (block 3, # blocks 1) of control file
ORA-00202: control file: '/SID/oradata/APEXP/control01.ctl'
ORA-27071: unable to seek to desired position in file
HP-UX Error: 9: Bad file number Additional information: 2

Cause of the Problem
The RMAN-06900, RMAN-06901 error occurs when the RMAN couldn't log the output in $rman_output. The actual investigation of the error comes in the associated additional error messages. If you look for additional error message we will see ORA-27071, HP-UX Error: 9: Bad file number. The word 'HP-UX Error: 9: Bad file number' indicates that error is coming from operating system. The OS user who is invoking RMAN don't have OS permission on the control file and hence HP-UX generates bad file number error.

Solution of the Problem
The user who is invoking RMAN don't have write permission on the controlfile. You must need to change permission from operating system. You can do so by issuing,
$chmod 666 /SID/oradata/APEXP/control01.ctl

After you change the permission now try to invoke RMAN command again.
Related Documents:

How to Restore the Controlfile from Backup.

ORA-00214: Controlfile Version Inconsistent on Startup or Shutdown

Controlfile in Oracle Database.

New Feature of 10.2g: Eliminate Control File Re-Creation

Creating controlfile fails with ORA-01503, ORA-01161

Backup Database control file -User Managed

Recover database after only lose of all controlfiles

Saturday, January 30, 2010

RMAN-06900, RMAN-06901, ORA-19921 maximum number of 64 rows exceeded

Problem Description
RMAN backups has been successfully completed but from the backup logs the following error have been generated.
Recovery Manager: Release 11.1.0.7.0 - Production on Fri Jan 29 00:01:15 2010

Copyright (c) 1982, 2007, Oracle. All rights reserved.

RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 64 rows exceeded

A variation of the above error is,
Recovery Manager: Release 11.1.0.7.0 - Production on Fri Jan 29 00:01:15 2010

Copyright (c) 1982, 2007, Oracle. All rights reserved.

RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 128 rows exceeded

Analysis And Solution of the Problem
As soon as you get error message RMAN-06900 and RMAN-06901 you immediately look for associated error messages. The associated error message should tell you more information and you need to look for those messages in order to solve this error. For example here we are getting additional error "ORA-19921: maximum number of 128 rows exceeded". So our solution will lie on ORA-19921.

RMAN gives a warning message of RMAN-6900, RMAN-6901 ORA-19921 when the output is too huge and oracle is not able to write the log output into v$rman_output. There are several reasons when oracle will not be able to write the log output into V$RMAN_OUTPUT. For example, one of your control file becomes corrupted, hence oracle is unable to write the the log output to RMAN view/table. The another major reasons cause this problem to happen is due to oracle bug.

You get oracle error "ORA-19921: maximum number of 64 rows exceeded" due to oracle Bug 465973.
You get oracle error "ORA-19921: MAXIMUM NUMBER OF 128 ROWS EXCEEDED" due to oracle Bug 8264365.

The bug 465973 is fixed in Oracle 10.2.0.4 and Oracle 11G. So if you see oracle only gives warning message and backup successfully done then you can simply ignore error messages or upgrade oracle or apply patch where bug is fixed.

However if you see your backup is not done due to these errors then immediately check for additional messages. If it happened due to controlfiles then take care of those errors.

Related Documents:

How to Restore the Controlfile from Backup.

ORA-00214: Controlfile Version Inconsistent on Startup or Shutdown

Controlfile in Oracle Database.

New Feature of 10.2g: Eliminate Control File Re-Creation

Creating controlfile fails with ORA-01503, ORA-01161

Backup Database control file -User Managed

Recover database after only lose of all controlfiles

Sunday, January 17, 2010

File Deletion Policy from Flash Recovery Area (FRA)

Flash Recovery Area (FRA) is used to set up a disk area where the database can create and manage a variety of files related to backup and recovery. If database level flashback is on then oracle creates flashback logs into flash recovery area. Also oracle creates archived logs in the flash recovery area. RMAN can store its backup sets and image copies in the flash recovery area too.

By default if flash recovery area is enabled then any backups of tablespace /datafile /database/ spfile/ controlfile/ archived logs go to flash recovery area. The site of flash recovery area is set by the parameter db_recovery_file_dest_size parameter and its location is specified by db_recovery_file_dest parameter. Now if the total file size inside flash recovery area is about to reach the it's limit then the files which to be deleted to reclaim space is determined by RMAN backup retention policy. Oracle Database does not delete eligible files from the flash recovery area until the space must be reclaimed for some other purpose. Thus, files recently moved to tape are often still available on disk for use in recovery. The recovery area can thus serve as a cache for tape. When the flash recovery area is full, Oracle Database automatically deletes eligible files to reclaim space in the recovery area as needed.

Based on the space in your flash recovery area oracle database will generate alert. The database issues a warning alert when reclaimable space is less than 15% and a critical alert when reclaimable space is less than 3%. When the recovery area is completely full and oracle sees there is no files for deletion from the flash recovery area, it will throw following errors,

ORA-19809: limit exceeded for recovery files
ORA-19804: cannot reclaim nnnn bytes disk space from mmmm limit


where nnnn is the number of bytes required and mmm is the disk quota.

So before you hit this error you should know which files from FRA should be eligible for deletion and which are not. I have made a list based on deletion policy.

1) Permanent files are never eligible for deletion. For example multiplexed redo log files and control files are part of database, that means they are permanent files and so they are never eligible for deletion.

2) Files that are obsolete under the retention policy are eligible for deletion. For example you set up CONFIGURE RETENTION POLICY TO REDUNDANCY 1 in RMAN configuration parameter but you have more than 1 copy backup and your FRA is under space pressure, then oracle can delete older copies of backup. In the same way if you configure RMAN to recovery window of 7 days and you have backups more than 7 days and those older backups are not needed to recover database in any point within 7 days then those older files than 7 days are eligible for deletion from FRA.

3) Transient files that have been copied to tape are eligible for deletion.

4) Archived redo logs are not eligible for deletion until all the consumers of the logs have satisfied their requirements ( If Archived Redo Log Deletion Policy is set ).
Note that, the default settings of archive log is
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
which mean by default archive logs are never eligible for deletion.

5) Foreign archived logs that have been mined by a LogMiner session on a logical standby database are eligible for deletion. Unlike an ordinary archived redo log, a foreign archived redo log has a different DBID.

6) If retention policy is set to none then no files will be eligible for deletion. But note that
if files are never considered obsolete, then a file can only be deleted from the flash recovery area if it has been backed up to some other disk location or to a tertiary storage device such as tape.

You can monitor flash recovery area space by following two views

A) V$RECOVERY_FILE_DEST : To find out the current location, disk quota, space in use, space reclaimable by deleting files, and total number of files in the flash recovery area.

B) V$FLASH_RECOVERY_AREA_USAGE : To find out the percentage of the total disk quota used by different types of files, how much space for each type of file can be reclaimed by deleting files that are obsolete, redundant, or already backed up to tape.

Related Documents
RMAN-06091 no channel allocated for maintenance (of an appropriate type)
RMAN-06429: TARGET database is not compatible with this version of RMAN
How to take RMAN backup on a remote disk/ location
RMAN-00554,RMAN-04005,ORA-0103 when remote connection by rman
Set Date format inside RMAN environment
How to skip a tablespace for restore operation
How to debug Backup, Restore Session in RMAN
Restore operation fails with RMAN-11003 ORA-01511 ORA-01516
Recover database after only lose of all controlfiles
RMAN-04014: startup failed: ORA-07446: sdnfy: bad value
How to restore an Spfile from autobackup older than 7 days
RMAN-06172: no autobackup found
Creating a Duplicate Database on a Remote Host -Part1
Database Duplication Fails Missing Log RMAN-06053 RMAN-06025
How to perform Database Point in time Recovery DBPITR

RMAN-06091 no channel allocated for maintenance (of an appropriate type)

Problem Description
RMAN DELETE OBSOLETE command fails with following error stack.
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of delete command at 01/17/2010 02:15:40
RMAN-06091: no channel allocated for maintenance (of an appropriate type)

Problem Investigation
This error occurs if you attempt to delete obsolete backup from tape but channel had not being allocated on tape. And also these error stacks will appear if you attempt to delete obsolete backup from disk but channel had not being allocation on disk.

Note that if you allocate channel to disk and then attempt to delete obsolete it should work if your backups are in disk. We assume that backups are in disk and let's try to delete it after allocating channel to disk.
RMAN> ALLOCATE CHANNEL FOR MAINTENANCE DEVICE TYPE DISK;

allocated channel: ORA_MAINT_DISK_2
channel ORA_MAINT_DISK_2: sid=149 devtype=DISK
RMAN> delete obsolete;

RMAN retention policy will be applied to the command
RMAN retention policy is set to redundancy 1
Deleting the following obsolete backups and copies:
Type Key Completion Time Filename/Handle
-------------------- ------ ------------------ --------------------
Backup Set 3739 Dec 24 2009 04:36:02
Backup Piece 6037 Dec 24 2009 04:36:02 lml1m82u_1_2
Backup Set 3738 Dec 24 2009 04:36:02
Backup Piece 6036 Dec 24 2009 04:36:02 lnl1m82u_1_2
Backup Set 3743 Dec 24 2009 04:38:06
Backup Piece 6038 Dec 24 2009 04:38:06 c-3269547898-20091224-01
.
.
.
Backup Set 3857 Jan 11 2010 04:25:21
Backup Piece 6217 Jan 11 2010 04:25:21 /db/app/oracle/product/10.2.0/db/dbs/ora_cfc-3269547898-2010 0111-00
Backup Set 3862 Jan 12 2010 04:58:50
Backup Piece 6223 Jan 12 2010 04:58:50 /db/app/oracle/product/10.2.0/db/dbs/ora_cfc-3269547898-2010 0112-00
Backup Set 3867 Jan 13 2010 04:40:29
Backup Piece 6228 Jan 13 2010 04:40:29 /db/app/oracle/product/10.2.0/db/dbs/ora_cfc-3269547898-2010 0113-00
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of delete command at 01/17/2010 02:15:40
RMAN-06091: no channel allocated for maintenance (of an appropriate type)
But no help. It fails still.

Solution of the Problem
Using show all command check the rman configuration parameter settings.
RMAN> show all;

RMAN configuration parameters are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1;
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK;
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO 'ora_cf%F';
CONFIGURE DEVICE TYPE DISK PARALLELISM 2 BACKUP TYPE TO BACKUPSET;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/db/app/oracle/product/10.2.0/db/dbs/snapcf_OMSRPS.f'; # default
From the above configuration we see that channels are allocated to disk.
Using the following command verify that whether the backup sets are on tape or in disk.
RMAN> list backup;

List of Backup Sets
===================

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
3743 Full 7.25M SBT_TAPE 00:01:55 Dec 24 2009 04:38:06
BP Key: 6038 Status: AVAILABLE Compressed: NO Tag: TAG20091224T043611
Handle: c-3269547898-20091224-01 Media:
Standby Control File Included: Ckp SCN: 1318360946 Ckp time: Dec 24 2009 04:32:46
SPFILE Included: Modification time: Aug 25 2009 03:53:48

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
3746 Incr 1 228.05M SBT_TAPE 00:08:49 Dec 25 2009 04:18:25
BP Key: 6045 Status: AVAILABLE Compressed: YES Tag: LEVEL_1_122509
Handle: lvl1ot00_1_2 Media:
List of Datafiles in backup set 3746
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- -------------------- ----
1 1 Incr 1320551570 Dec 25 2009 03:40:48 /db/app/oradata/OMSRPS/system01.dbf
2 1 Incr 1320551570 Dec 25 2009 03:40:48 /db/app/oradata/OMSRPS/undotbs01.dbf
3 1 Incr 1320551570 Dec 25 2009 03:40:48 /db/app/oradata/OMSRPS/sysaux01.dbf
7 1 Incr 1320551570 Dec 25 2009 03:40:48 /db/app/oradata/OMSRPS/rman_ts.dbf

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
3747 Incr 1 136.52M SBT_TAPE 00:17:24 Dec 25 2009 04:27:00
BP Key: 6044 Status: AVAILABLE Compressed: YES Tag: LEVEL_1_122509
Handle: lul1ot00_1_2 Media:
List of Datafiles in backup set 3747
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- -------------------- ----
4 1 Incr 1320551570 Dec 25 2009 03:40:48 /db/app/oradata/OMSRPS/users01.dbf
5 1 Incr 1320551570 Dec 25 2009 03:40:48 /db/app/oradata/OMSRPS/mgmt.dbf
6 1 Incr 1320551570 Dec 25 2009 03:40:48 /db/app/oradata/OMSRPS/mgmt_ecm_depot1.dbf
From the above command we see the backups are on SBT_TAPE. So you have to issue delete command from the tape.

So our command will be as follows.
RMAN> allocate channel for maintenance device type 'sbt_tape' PARMS '...';
Here please change '...' to your actual tape params

RMAN> delete obsolete;

or issue following command to delete obsolete backups from tape.
RMAN> allocate channel for maintenance type sbt_tape;

However if your backups are on disk then your command will look like below.
RMAN> allocate channel for maintenance type disk;

allocated channel: ORA_MAINT_DISK_4
channel ORA_MAINT_DISK_4: sid=79 devtype=DISK
And then,
RMAN> delete obsolete device type disk;

RMAN retention policy will be applied to the command
RMAN retention policy is set to redundancy 1
Deleting the following obsolete backups and copies:
Type Key Completion Time Filename/Handle
-------------------- ------ ------------------ --------------------
Backup Set 3792 Jan 01 2010 04:25:18
Backup Piece 6115 Jan 01 2010 04:25:18 /db/app/oracle/product/10.2.0/db/dbs/ora_cfc-3269547898-2010 0101-00
Backup Set 3799 Jan 02 2010 04:25:44
Backup Piece 6126 Jan 02 2010 04:25:44 /db/app/oracle/product/10.2.0/db/dbs/ora_cfc-3269547898-2010 0102-00
.
.
.
Backup Piece 6217 Jan 11 2010 04:25:21 /db/app/oracle/product/10.2.0/db/dbs/ora_cfc-3269547898-2010 0111-00
Backup Set 3862 Jan 12 2010 04:58:50
Backup Piece 6223 Jan 12 2010 04:58:50 /db/app/oracle/product/10.2.0/db/dbs/ora_cfc-3269547898-2010 0112-00
Backup Set 3867 Jan 13 2010 04:40:29
Backup Piece 6228 Jan 13 2010 04:40:29 /db/app/oracle/product/10.2.0/db/dbs/ora_cfc-3269547898-2010 0113-00

Do you really want to delete the above objects (enter YES or NO)? yes
deleted backup piece
backup piece handle=/db/app/oracle/product/10.2.0/db/dbs/ora_cfc-3269547898-20100101-00 recid=6115 stamp=7071 13518
deleted backup piece
backup piece handle=/db/app/oracle/product/10.2.0/db/dbs/ora_cfc-3269547898-20100102-00 recid=6126 stamp=7071 99944
deleted backup piece
.
.
.
deleted backup piece
backup piece handle=/db/app/oracle/product/10.2.0/db/dbs/ora_cfc-3269547898-20100112-00 recid=6223 stamp=7080 65929
deleted backup piece
backup piece handle=/db/app/oracle/product/10.2.0/db/dbs/ora_cfc-3269547898-20100113-00 recid=6228 stamp=7081 51228
Deleted 13 objects

If you want to delete both obsolete and expired backups from disk then you can issue,
RMAN> ALLOCATE CHANNEL FOR MAINTENANCE DEVICE TYPE DISK;
RMAN> run
{
report obsolete;
CROSSCHECK BACKUP;
CROSSCHECK COPY;
DELETE EXPIRED BACKUP;
DELETE EXPIRED COPY;
delete obsolete;
}
release channel;
Related Documents
RMAN-06091 no channel allocated for maintenance (of an appropriate type)
RMAN-06429: TARGET database is not compatible with this version of RMAN
How to take RMAN backup on a remote disk/ location
RMAN-00554,RMAN-04005,ORA-0103 when remote connection by rman
Set Date format inside RMAN environment
How to skip a tablespace for restore operation
How to debug Backup, Restore Session in RMAN
Restore operation fails with RMAN-11003 ORA-01511 ORA-01516
Recover database after only lose of all controlfiles
RMAN-04014: startup failed: ORA-07446: sdnfy: bad value
How to restore an Spfile from autobackup older than 7 days
RMAN-06172: no autobackup found
Creating a Duplicate Database on a Remote Host -Part1
Database Duplication Fails Missing Log RMAN-06053 RMAN-06025
How to perform Database Point in time Recovery DBPITR

Thursday, April 16, 2009

RMAN-06429: TARGET database is not compatible with this version of RMAN

Whenever you connect to a database (rman version if different from source database version in fact rman version is higher than the source database) through RMAN using/without recovery catalog it fails with RMAN-06429: TARGET database is not compatible with this version of RMAN as below.

C:\>rman target rman/rman@local2 catalog rman/rman@catdb

Recovery Manager: Release 10.2.0.1.0 - Production on Tue Mar 24 05:07:14 2009

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Recovery Manager incompatible with TARGET database: RMAN 8.0.4.0 to 10.1.0.0 req
uired
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00554: initialization of internal recovery manager package failed
RMAN-06429: TARGET database is not compatible with this version of RMAN

Cause of the Problem
You have higher RMAN version than the target database (to which you are connecting). In order to successfully connect to database by rman the rman version must be less or equal to the target database version.

Solution of the Problem
There is a rule of thumb in this connecting issue and it is best practice to use 'rule of thumb'. That is "always use the target database RMAN executable and the latest release of Oracle for your recovery catalog."

However, if you are going to use a lower version RMAN executable to backup higher
version databases then remember you are restricted to only the features available
in the database your using. But the executable you are using must be able to
handle the feature or configuration you are using.

In the above example a 10.2.0.1 rman executable tried to connect to database version 10.1.0.2 which is not supported. If we follow rule of thumb then also use 10.1.0.2 rman executable to connect to database 10.1.0.2 in order to best use of all the available features. However we can connect to 10.1.0.2 database with lower version of rman executable but that is not recommended as in that case we might be restricted to use less features.

One more thing is clear from the error message is,
"Recovery Manager incompatible with TARGET database: RMAN 8.0.4.0 to 10.1.0.0
required". So you can use any rman executable to connect to target database (which- the target database in fact 10.1g) but current RMAN executable is shown as 10.2.0.1.

Related Documents
http://arjudba.blogspot.com/2008/10/rman-00554rman-04005ora-0103-when.html
http://arjudba.blogspot.com/2008/08/rman-06426-recoverycatalogowner-role.html

Sunday, October 12, 2008

How to take RMAN backup on a remote disk/ location

It is common to hear that for backup there is not sufficient space on my server or my server disk is about to full. And now I want to take my RMAN backup directly to remote location. How I can achieve that?

Now before taking backup let think in which type of storage on remote location you want to take backup? Is it a disk or to the tape. Below is the procedure for both types.

Taking Remote RMAN backup to Tape
1)This is of no different as you access remote tape devices.

2)First install and configures the drivers of the tape device on the local machine box.

3)Make sure that you can access to tape by making copies file to tape.

4)If step3 is successful then RMAN will be able to take backup to tape directly.

Taking Remote RMAN backup to disk
1)Remember that direct disk backup to remote location using RMAN is not possible as we can do it on local disk by using FORMAT parameter specifying local disk.

2)As RMAN backups to DISK can only be taken to some 'part' of the target database server. So, remote location will not work.

3)The thing you can achieve goal is to share remote location, then mount it on the target database where you want to take backup. An example of how to share folder and access it on other machine is discussed on How to share and Access file on Solaris Machine

4)Take RMAN backup to this mapped location using FORMAT clause. Based on this a remote backup is taken in http://arjudba.blogspot.com/2008/10/backup-to-remote-location-fails-with.html after sharing remote location and map it to location /export/home/oracle/remote/.
Related Documents
RMAN Backup format specification
Set up a Database for RMAN Backup
Create Multiple Duplex backup by one Backup Command

Wednesday, October 8, 2008

RMAN-00554,RMAN-04005,ORA-0103 when remote connection by rman

Whenever I try to connect remotely to a database through rman it fails with message RMAN-00554, RMAN-04005, ORA-0103.
In this example my source database is saturn and destination database is jupiter. From jupiter let me try to connect to saturn.
Source database
Before proceed let's see the parameter that we can use to connect to rman without tnsnames.ora. Our needed parameter is hostname/ipaddress, service_name and port number. These three can be found by issuing lsnrctl status in source machine.
Note that you must have a password file setup in the source database. This requires the use of the "orapwd" command and the initialization parameter "REMOTE_LOGIN_PASSWORDFILE".

bash-3.00$ hostname
saturn
SQL> show parameter remote_login
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
remote_login_passwordfile string EXCLUSIVE

bash-3.00$ lsnrctl status

LSNRCTL for Solaris: Version 10.2.0.1.0 - Production on 08-OCT-2008 06:07:54

Copyright (c) 1991, 2005, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Solaris: Version 10.2.0.1.0 - Production
Start Date 27-JUL-2008 02:00:31
Uptime 73 days 4 hr. 7 min. 23 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /oracle/app/oracle/product/10.2.0/db_1/network/admin/listener.ora
Listener Log File /oracle/app/oracle/product/10.2.0/db_1/network/log/listener.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=saturn)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "ARJU.SATURN.ARJUBD.COM" has 1 instance(s).
Instance "ARJU", status READY, has 1 handler(s) for this service...
The command completed successfully

Our interest is bolded. hostname=saturn, port=1521(default) and service_name=ARJU.SATURN.ARJUBD.COM
Try to connect to source database from destination database (from jupiter to saturn)
bash-3.00$ hostname
jupiter

bash-3.00$ rman target system/sistem@saturn/ARJU.SATURN.ARJUBD.COM

Recovery Manager: Release 10.2.0.1.0 - Production on Wed Oct 8 05:48:50 2008

Copyright (c) 1982, 2005, Oracle. All rights reserved.

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00554: initialization of internal recovery manager package failed
RMAN-04005: error from target database:
ORA-01031: insufficient privileges

We did not used port here as port is default. If it was not default suppose it was 1522 then we would use rman target system/sistem@saturn:1522/ARJU.SATURN.ARJUBD.COM
Diagnosis why problem happens (In the source database)
In order to connect to rman a user must have the sysdba system privilege otherwise it will fail. As we are login through network password file authentication will be used. And in the password file user system must be entryed as sysdba.

SQL> !hostname
saturn
SQL> select * from v$pwfile_users;

USERNAME SYSDB SYSOP
------------------------------ ----- -----
SYS TRUE TRUE

As we see here in the passwordfile only SYS is there. So only sys can do the task to connect to rman through network. In order to permit system to logon through network grant him to SYSDBA.

SQL> grant sysdba to system;
Grant succeeded.

Now check the entry in passwordfile.
SQL> select * from v$pwfile_users;
USERNAME SYSDB SYSOP
------------------------------ ----- -----
SYS TRUE TRUE
SYSTEM TRUE FALSE


Check now from destination database (Here it is jupiter)

bash-3.00$ hostname
jupiter
bash-3.00$ rman target system/sistem@saturn/ARJU.SATURN.ARJUBD.COM

Recovery Manager: Release 10.2.0.1.0 - Production on Wed Oct 8 05:50:13 2008

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to target database: ARJU (DBID=2869417476)

And now it succeeds.

Related Documents
RMAN-06426: RECOVERY_CATALOG_OWNER role must be granted to user
Working with RMAN stored scripts in Recovery Catalog
Set up a Database for RMAN Backup

Sunday, September 21, 2008

Set Date format inside RMAN environment

Suppose inside rman environment I got the following output.
RMAN> list backup;

using target database control file instead of recovery catalog

List of Backup Sets
===================

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
1 Full 558.80M DISK 00:01:14 08-SEP-08
BP Key: 1 Status: AVAILABLE Compressed: NO Tag: TAG20080908T035835
Piece Name: /oracle/app/oracle/product/10.2.0/db_1/flash_recovery_area/ARJU/backupset/
2008_09_08/o1_mf_nnndf_TAG20080908T035835_4d9pscwz_.bkp
List of Datafiles in backup set 1
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
1 Full 1030282 08-SEP-08 /oradata2/arjudba/arjudba/arju/system01.dbf
2 Full 1030282 08-SEP-08 /oradata2/arjudba/arjudba/arju/undotbs01.dbf
3 Full 1030282 08-SEP-08 /oradata2/arjudba/arjudba/arju/sysaux01.dbf
4 Full 1030282 08-SEP-08 /oradata2/arjudba/arjudba/arju/users01.dbf

.
.
.

Here I get date as 08-SEP-08 which only shows date, month and year. The exact minute, hour and second are not displayed here. In order to get output as hour,minute second along with above output I have to set NLS_DATE_FORMAT. Note that this need to be set in OS level. Suppose in my bash shell, I have set
-bash-3.00$ export NLS_DATE_FORMAT='DD-MON-YY HH24:MI:SS'
Ans now it returns as I wanted.
RMAN> list backup;

using target database control file instead of recovery catalog

List of Backup Sets
===================

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ------------------
1 Full 558.80M DISK 00:01:14 08-SEP-08 03:59:49
BP Key: 1 Status: AVAILABLE Compressed: NO Tag: TAG20080908T035835
Piece Name: /oracle/app/oracle/product/10.2.0/db_1/flash_recovery_area/ARJU/backupset/
2008_09_08/o1_mf_nnndf_TAG20080908T035835_4d9pscwz_.bkp
List of Datafiles in backup set 1
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- ------------------ ----
1 Full 1030282 08-SEP-08 03:58:35 /oradata2/arjudba/arjudba/arju/system01.dbf
2 Full 1030282 08-SEP-08 03:58:35 /oradata2/arjudba/arjudba/arju/undotbs01.dbf
3 Full 1030282 08-SEP-08 03:58:35 /oradata2/arjudba/arjudba/arju/sysaux01.dbf
4 Full 1030282 08-SEP-08 03:58:35 /oradata2/arjudba/arjudba/arju/users01.dbf

On windows you have to set as,
>set NLS_DATE_FORMAT=MON DD, YYYY HH24:MI:SS

Monday, September 15, 2008

How to skip a tablespace for restore operation

Sometimes we want to restore full database except a certain tablespace. That is we need to skip a tablespace for restore operation. May be that the tablespace only contains tempoarary data and hence not useful for us. Or we are performing some database point in time recovery operation and we want to omit a big tablespace as the tablespace does not contain data needed for our requirement. In this way we can minimize our restore operation time.

With the RESTORE DATABASE command we perform all datafiles restore operation except those that are offline or read-only.

Note that RESTORE DATABASE does not work same as BACKUP DATABASE. With command BACKUP DATABASE, RMAN backs up datafiles along with controlfiles and spfile. But with RESTORE COMMAND operation, RMAN only restores datafiles.

To omit a certain tablespace for restore operation use RESTORE DATABASE SKIP TABLESPACE tablespace_name. Suppose I want to omit restore of indexed tablespace INDX01,INDX02 and INDX01_16K . Then my restore command will be,
RMAN>RESTORE DATABASE SKIP TABLESPACE INDX01,INDX02,INDX01_16K;

If you specify SKIP FOREVER TABLESPACE, then RMAN specifies the DROP option of ALTER DATABASE DATAFILE ... OFFLINE when taking the datafiles that belong to the tablespace offline before the restore. The DROP option indicates that RMAN does not intend to recover these files and intends to drop their tablespaces from the database after the database is opened again. In other words, FOREVER indicates that RMAN never intends to do anything with the skipped tablespaces again.

Suppose you want to skip forever to restore tablespace EXAMPLE,INDX01 and INDX02 then your command will be,
RMAN>RESTORE DATABASE SKIP FOREVER TABLESPACE EXAMPLE, INDX01, INDX02;

Monday, September 8, 2008

How to debug Backup, Restore Session in RMAN

In many cases in RMAN we need to debug the session to find and investigate the session about what is happening there. Also to obtain and check the correct diagnostic evidence we sometimes need the debug the rman session.

The most common use of debugging RMAN session is whenever we fail RMAN operation. In fact if any failure in RMAN operation RMAN log and debug files are NOT generated by default and we explicitly need to enable it.

We can log and debug session whichever operation we do in RMAN. Like Backup, Restore, TSPITR, DBPITR, Duplicate Database, Restore to a new host etc.

The commnad is simple,
The syntax is,
$ rman target [un/pwd@target_db] catalog [un/pwd@catalog_db] debug trace rman.trc log rman.log
RMAN>[RMAN Commands Here]


Where un indicates username and pwd indicates password.

In my system I used debugging and logging RMAN session by,
$ rman target / log=/backup03/webkey/rmanlog.txt trace=/backup03/webkey/rmantrace.log
RMAN> debug on
RMAN> @/backup03/webkey/rman_script
RMAN> debug off;
RMAN> exit;


Where my restore commands was inside the /backup03/webkey/rman_script file. And don't confuse with the sequence 2,3,4 here printed. It is line number inside my script of backup.

Later we can see the log information from file rmanlog.txt by,
$ less /backup03/webkey/rmanlog.txt (On Unix System)

We can see debug information from rmantrace.log by,
$ less /backup03/webkey/rmantrace.log (On my Unix System)
.
.
.
.

DBGMISC: command to be compiled and executed is: DEBUG [03:13:27.795] (krmice)
DBGMISC: command after this command is: NONE [03:13:27.795] (krmice)
DBGMISC: current incarnation does not matter for DEBUG [03:13:27.795] (krmice)
Debugging turned off

Above is the sample output of the file of /backup03/webkey/rmantrace.log

Sunday, September 7, 2008

Restore operation fails with RMAN-11003 ORA-01511 ORA-01516

Error Description And Symptoms
I am performing disaster recovery operation. The new host path are different than from original host. So for the datafile I used SET NEWNAME .. clause. And for the online redo log file I used SQL "ALTER DATABASE RENAME FILE .. " clause to rename the online logfiles to a valid location on to a new host. I have already ran the restore srcipt in previous but it fails to rename third online redo logfiles due to invalid location. And after fixing location whenever I ran the script again it fails with serveral RMAN and Oracle error as below.
RMAN> @/backup03/webkey/rman_script

RMAN> run{
2> set newname for datafile 1 to '/backup03/webkey/system01.dbf';
3> set newname for datafile 2 to '/backup03/webkey/undotbs01.dbf';
4> set newname for datafile 3 to '/backup03/webkey/sysaux01.dbf';
5> set newname for datafile 4 to '/backup03/webkey/users01.dbf';
6> set newname for datafile 5 to '/backup03/webkey/data01.dbf';
7> set newname for datafile 6 to '/backup03/webkey/data02.dbf';
8> set newname for datafile 7 to '/backup03/webkey/indx01.dbf';
9> set newname for datafile 8 to '/backup03/webkey/indx02.dbf';
10> set newname for datafile 9 to '/backup03/webkey/tools01.dbf';
11> set newname for datafile 10 to '/backup03/webkey/splex_data.dbf';
12> set newname for datafile 11 to '/backup03/webkey/splex_indx.dbf';
13> set newname for datafile 12 to '/backup03/webkey/columnbia.dbf';
14> set newname for datafile 13 to '/backup03/webkey/data03.dbf';
15> set newname for datafile 14 to '/backup03/webkey/data03b.dbf';
16> set newname for datafile 15 to '/backup03/webkey/data_01_16K_01.dbf';
17> set newname for datafile 16 to '/backup03/webkey/indx_01_16K_01.dbf';
18> SQL "ALTER DATABASE RENAME FILE ''/d00/oradata/db101db/redo01a.log''
19> TO ''/backup03/webkey/redo01a.log'' ";
20> SQL "ALTER DATABASE RENAME FILE ''/d01/oradata/db101db/redo01b.log''
21> TO ''/backup03/webkey/redo01b.log'' ";
22> SQL "ALTER DATABASE RENAME FILE ''/d02/oradata/db101db/redo01c.log''
23> TO ''/backup03/webkey/redo01c.log'' ";
24> SQL "ALTER DATABASE RENAME FILE ''/d00/oradata/db101db/redo02a.log''
25> TO ''/backup03/webkey/redo02a.log'' ";
26> SQL "ALTER DATABASE RENAME FILE ''/d01/oradata/db101db/redo02b.log''
27> TO ''/backup03/webkey/redo02b.log'' ";
28> SQL "ALTER DATABASE RENAME FILE ''/d02/oradata/db101db/redo02c.log''
29> TO ''/backup03/webkey/redo02c.log'' ";
30> SQL "ALTER DATABASE RENAME FILE ''/d00/oradata/db101db/redo03a.log''
31> TO ''/backup03/webkey/redo03a.log'' ";
32> SQL "ALTER DATABASE RENAME FILE ''/d01/oradata/db101db/redo03b.log''
33> TO ''/backup03/webkey/redo03b.log'' ";
34> SQL "ALTER DATABASE RENAME FILE ''/d02/oradata/db101db/redo03c.log''
35> TO ''/backup03/webkey/redo03c.log'' ";
36> SET UNTIL TIME '06-JUN-08 15:15:00';
37> RESTORE DATABASE;
38> SWITCH DATAFILE ALL;
39> RECOVER DATABASE;
40> }
executing command: SET NEWNAME
using target database control file instead of recovery catalog

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

sql statement: ALTER DATABASE RENAME FILE ''/d00/oradata/db101db/redo01a.log'' TO ''/backup03/webkey/redo01a.log''
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of sql command on default channel at 09/08/2008 02:30:42
RMAN-11003: failure during parse/execution of SQL statement: ALTER DATABASE RENAME FILE '/d00/oradata/db101db/redo01a.log' TO '/backup03/webkey/redo01a.log'
ORA-01511: error in renaming log/data files
ORA-01516: nonexistent log file, datafile, or tempfile "/d00/oradata/db101db/redo01a.log"


In previous ran of this script it could rename successfully of the first two online redo logfile and it is recorded on the controlfile but the restore operation fails. Now subsequent run of the script will fail because no log file actually exist in the controlfile as they already renamed.

In previous run m,y online redo logfile is rename from /d00/oradata/db101db/redo01a.log to /backup03/webkey/redo01a.log. In the next run(in this example) it again try to search for online redo log in the location /d00/oradata/db101db/redo01a.log inside the controlfile but did not find any as they already modified. So error appear of nonexistent log file, datafile, or tempfile.

Solution of The Problem
While make database in mount state log on to SQL*Plus and query the member column of v$logfile and see the location of the member. Suppose only 1 member is modified then in the SQL*Plus you can issue,
SQL>ALTER DATABASE RENAME FILE '/backup03/webkey/redo01a.log' TO '/d00/oradata/db101db/redo01a.log';
to rename reflected new location to old one and thus you can re-run the script without modifying any line.

If you notice that are redolog files are renamed successfully to the new location then you can simply omit/comment the lines of rename redo log command. Like in below.
tdev> vi /backup03/webkey/rman_script
run{
set newname for datafile 1 to '/backup03/webkey/system01.dbf';
set newname for datafile 2 to '/backup03/webkey/undotbs01.dbf';
set newname for datafile 3 to '/backup03/webkey/sysaux01.dbf';
set newname for datafile 4 to '/backup03/webkey/users01.dbf';
set newname for datafile 5 to '/backup03/webkey/data01.dbf';
set newname for datafile 6 to '/backup03/webkey/data02.dbf';
set newname for datafile 7 to '/backup03/webkey/indx01.dbf';
set newname for datafile 8 to '/backup03/webkey/indx02.dbf';
set newname for datafile 9 to '/backup03/webkey/tools01.dbf';
set newname for datafile 10 to '/backup03/webkey/splex_data.dbf';
set newname for datafile 11 to '/backup03/webkey/splex_indx.dbf';
set newname for datafile 12 to '/backup03/webkey/columnbia.dbf';
set newname for datafile 13 to '/backup03/webkey/data03.dbf';
set newname for datafile 14 to '/backup03/webkey/data03b.dbf';
set newname for datafile 15 to '/backup03/webkey/data_01_16K_01.dbf';
set newname for datafile 16 to '/backup03/webkey/indx_01_16K_01.dbf';
# SQL "ALTER DATABASE RENAME FILE ''/d00/oradata/db101db/redo01a.log''
# TO ''/backup03/webkey/redo01a.log'' ";
# SQL "ALTER DATABASE RENAME FILE ''/d01/oradata/db101db/redo01b.log''
# TO ''/backup03/webkey/redo01b.log'' ";
# SQL "ALTER DATABASE RENAME FILE ''/d02/oradata/db101db/redo01c.log''
# TO ''/backup03/webkey/redo01c.log'' ";
# SQL "ALTER DATABASE RENAME FILE ''/d00/oradata/db101db/redo02a.log''
# TO ''/backup03/webkey/redo02a.log'' ";
# SQL "ALTER DATABASE RENAME FILE ''/d01/oradata/db101db/redo02b.log''
# TO ''/backup03/webkey/redo02b.log'' ";
# SQL "ALTER DATABASE RENAME FILE ''/d02/oradata/db101db/redo02c.log''
# TO ''/backup03/webkey/redo02c.log'' ";
# SQL "ALTER DATABASE RENAME FILE ''/d00/oradata/db101db/redo03a.log''
# TO ''/backup03/webkey/redo03a.log'' ";
# SQL "ALTER DATABASE RENAME FILE ''/d01/oradata/db101db/redo03b.log''
# TO ''/backup03/webkey/redo03b.log'' ";
# SQL "ALTER DATABASE RENAME FILE ''/d02/oradata/db101db/redo03c.log''
# TO ''/backup03/webkey/redo03c.log'' ";

SET UNTIL TIME '06/06/2008 15:15:00';
RESTORE DATABASE;
SWITCH DATAFILE ALL;
RECOVER DATABASE;
}
"/backup03/webkey/rman_script" 41 lines, 2179 characters

And then run the script again.

Monday, September 1, 2008

On Solaris 64 bit Rman fails with libclntsh.so.10.1, ld.so.1, Segmentation Fault (core dumped)

Problem Description and Solution Step by Step
1.
Whenever you try to connect to rman then it fails with ld.so.1: rman: fatal: libclntsh.so.10.1: open failed: No such file or directory like,
bash-3.00$ rman target /
ld.so.1: rman: fatal: libclntsh.so.10.1: open failed: No such file or directory
Killed

2.
But you browse the directory $ORACLE_HOME/lib32 and see libclntsh.so.10.1 is present and also has the correct permission on it.
bash-3.00$ ls -l $ORACLE_HOME/lib32/libclntsh.so.10.1
-rwxr-xr-x 1 oracle oinstall 24094536 Apr 17 12:35 /oracle/app/oracle/product/10.2.0/db_1/lib32/libclntsh.so.10.1

3.
The missing thing is your LD_LIBRARY_PATH is not set. So I have set it below on my bash shell. If you use another shell then use shell specific command to set the environmental variable. On C shell use setenv LD_LIBRARY_PATH=$ORACLE_HOME/lib32. If you are on windows machine then use
set LD_LIBRARY_PATH=$ORACLE_HOME\lib32
As I am on bash shell on solaris so I set it as,
bash-3.00$ export LD_LIBRARY_PATH=$ORACLE_HOME/lib32

4.
Now try to connect rman and see it fails with rong ELF class: ELFCLASS32.
bash-3.00$ rman target /
ld.so.1: rman: fatal: /oracle/app/oracle/product/10.2.0/db_1/lib32/libclntsh.so.10.1: wrong ELF class: ELFCLASS32
Killed

5.
As my Solaris is 64 bit so I need to set LD_LIBRARY_PATH_64 environmental variable also.
bash-3.00$ export LD_LIBRARY_PATH_64=$ORACLE_HOME/lib

6.
You can easily avoid every time setting of these export or setenv or set command on your system each time by editing the profile or setting environmental variable on windows and make it permanent.

7.
On Solaris machine it is .profile
If you use linux bash shell then it is .bash_profile on home directory.

8.
I modified .profile file and added two lines in it like below
-bash-3.00$ vi .profile
export LD_LIBRARY_PATH=$ORACLE_HOME/lib32
export LD_LIBRARY_PATH_64=$ORACLE_HOME/lib


9.
To make the immediate affect of these changes use source command like,
-bash-3.00$ source .profile

10.
Now try to connect to your database.
bash-3.00$ rman target /

Recovery Manager: Release 10.2.0.1.0 - Production on Tue Sep 2 02:25:13 2008
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Segmentation Fault (core dumped)

11.
Oh, you still fails. However I restarted the server machine and try to connect to rman and then see errors goes away and connect to rman.
I did it

bash-3.00$ su -
Password:
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
# reboot

12.
Then after starting up I issued rman command.
bash-3.00$ rman target /

Recovery Manager: Release 10.2.0.1.0 - Production on Tue Sep 2 03:46:31 2008

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to target database: ARJU (DBID=2871324595)

Sunday, August 31, 2008

Recover database after only lose of all controlfiles

This example is based on,
You have lost all your current and backup of controlfile.
You have avaiable your current data files.
You have available your online redo log files.


In the scenario, I will show the procedure of how to recover database if you lose your all copies of controlfiles and you don't have any backup of your controlfile.

Let's start by deleting controlfile of my running database.

1.In order to know the controlfiles of my database issue,
SQL> select name from v$controlfile;

NAME
--------------------------------------------------------------------------------
/oradata2/arjudba/arjudba/control01.ctl
/oradata2/arjudba/arjudba/control02.ctl
/oradata2/arjudba/arjudba/control03.ctl

2.Delete all copies of controlfile. Here with one command I have deleted. But before deleting with one such command be conscious that this command will not delete other important files. You can delete files one by one.
SQL> !rm /oradata2/arjudba/arjudba/control0*

3.Now let's see whether controlfile is available or not by issuing following command.
SQL> shutdown abort
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 314572800 bytes
Fixed Size 2020416 bytes
Variable Size 121637824 bytes
Database Buffers 184549376 bytes
Redo Buffers 6365184 bytes
ORA-00205: error in identifying control file, check alert log for more info

As it can't read controlfile so ORA-00205 arises.

4.From this state is your situation what you need to do. If you have previously issued ALTER DATABASE BACKUP CONTROLFILE TO TRACE then use that file and modify. If you don't have any backup of controlfile then you have to create a new one as below. Note that you have to remember the name of datafile and online redo log file.
-bash-3.00$ vi /oradata2/ctl.ctl
STARTUP NOMOUNT
CREATE CONTROLFILE REUSE DATABASE "ARJUDBA" NORESETLOGS ARCHIVELOG
MAXLOGFILES 16
MAXLOGMEMBERS 3
MAXDATAFILES 100
MAXINSTANCES 8
MAXLOGHISTORY 292
LOGFILE
GROUP 1 '/oradata2/arjudba/arjudba/redo01.log' SIZE 50M,
GROUP 2 '/oradata2/arjudba/arjudba/redo02.log' SIZE 50M,
GROUP 3 '/oradata2/arjudba/arjudba/redo03.log' SIZE 50M
DATAFILE
'/oradata2/arjudba/arjudba/system01.dbf',
'/oradata2/arjudba/arjudba/undotbs01.dbf',
'/oradata2/arjudba/arjudba/sysaux01.dbf',
'/oradata2/arjudba/arjudba/users01.dbf'
CHARACTER SET WE8ISO8859P1
;

5.Save the script and run it inside SQL*plus.
SQL> @/oradata2/ctl.ctl
ORA-01081: cannot start already-running ORACLE - shut it down first

Control file created.

6. At this stage your database is mounted. You need to recover it.
SQL> recover database;
Media recovery complete.

7.Open the database after recovery completes.
SQL> alter database open;
Database altered.

8.As Create controlfile statement does not include Temp tablespace you may need to add it.
SQL>ALTER TABLESPACE TEMP ADD TEMPFILE '/oradata2/arjudba/arjudba/temp01.dbf'
SIZE 20971520 REUSE AUTOEXTEND ON NEXT 655360 MAXSIZE 32767M;


Related Documents
Recover database after missing online redo logs and all controlfiles.
Purpose and Restriction of Recover Command in Oracle
Restore and Recover database in Noarchivelog Mode

Thursday, August 28, 2008

RMAN-04014: startup failed: ORA-07446: sdnfy: bad value

Problem Description
While performing disaster recovery to a new host, you need to start the database in nomount sate(in fact to restore control file) but database could not start in nomount state because it fails with RMAN-04014, ORA-07446.

RMAN> startup force pfile='/oradata2/arjudbapfile.ora';

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of startup command at 08/28/2008 15:03:44
RMAN-04014: startup failed: ORA-07446: sdnfy: bad value '/oracle/app/oracle/product/10.2.0/db_1/admin/arjudba/udump' for parameter user_dump_dest.

Cause of The Problem
The path /oracle/app/oracle/product/10.2.0/db_1/admin/arjudba/udump as shown in output does not exist. Oracle itself does not create any path if a path does not exist. So, you have to change the value of user_dump_dest in the initialize parameter.

Solution of The Problem
If you use pfile to start your database then edit the pfile with any editor (for example vi on unix) and either change the location of user_dump_dest or remove the parameter user_dump_dest from pfile. And then perform startup.


If you like to use spfile then first create pfile from spfile, then edit the pfile and then create spfile from the pfile, like.

1.Create pfile from spfile.
SQL> create pfile='1.pfile' from spfile='/oracle/app/oracle/product/10.2.0/db_1/dbs/spfilearju.ora';

2.Edit the pfile.
SQL>!vi 1.pfile

Remove the entry of user_dump_dest or add valid entry.

3.Create spfile from pfile.
SQL>create spfile from pfile='1.pfile';

4.start the database.
SQL>STARTUP NOMOUNT

Message file RMAN.msb not found

Error Description
I give the full path of rman executable file location and I am getting error RMAN<>.msb not found as below.
-bash-3.1$ /oradata2/bin/rman target /
Message file RMAN<>.msb not found

Verify that ORACLE_HOME is set properly

Solution of The problem
Believe me, as I still got you have not set ORACLE_HOME properly. So set it.

To know your current settings of ORACLE_HOME, issue,
-bash-3.1$ echo $ORACLE_HOME
/oradata2/bin/

Here we see it is set, it may either unset. Though it is set wrong. The ORACLE_HOME path is before the bin directory. So here ORACLE_HOME will be /oradata2 instead of /oradata2/bin/

On unix set the value for the current session by,
-bash-3.1$ export ORACLE_HOME=/oradata2/

On Windows environment you have to set by
set ORACLE_HOME=C:\oracle or like that.

In order to set it permanently edit your profile. On linux like, ~/.bash_profile or on unix edit the file .profile on home directory and make an entry of ORACLE_HOME.

After setting correct ORACLE_HOME now try to connect to rman.
-bash-3.1$ /oradata2/bin/rman target /

Recovery Manager: Release 10.2.0.1.0 - Production on Thu Aug 28 14:37:53 2008

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to target database (not started)