New features in Oracle 11g Data Guard

Oracle 11g already have two releases. Oracle 11g Release 1 (11.1g) and Oracle 11g Release 2(11.2g). In this post I will specify new features available to Oracle data guard 11g separately.

New Features in Oracle Data Guard 11.1 (Applicable to both Redo Apply and SQL Apply)
- The COMPRESSION attribute is used to specify whether redo data is compressed before transmission to a redo transport destination.

- With NET_TIMEOUT attribute it is specified the number of seconds that the LGWR background process will block waiting for a redo transport destination to acknowledge redo data sent to it. If an acknowledgement is not received within NET_TIMEOUT seconds, an error is logged and the redo transport session to that destination is terminated.

- Role transitions happened faster than previous versions.

- Strong authentication for redo transport network sessions.

- Simplified Data Guard management interface by deprecation of redundant SQL clauses and initialization parameters.

- A physical standby database can now take advantage of the rolling upgrade feature provided by a logical standby. Through the use of the new KEEP IDENTITY clause option to the SQL ALTER DATABASE RECOVER TO LOGICAL STANDBY statement, a physical standby database can be temporarily converted into a logical standby database for the rolling upgrade, and then reverted back to the original configuration of a primary database and a physical standby database when the upgrade is done.

- In the same Data Guard configuration now it allows a mix of Linux and Windows primary and standby databases.

New Features in Oracle Data Guard 11.1 (Applicable to only Redo Apply)
- You can open ans query physical standby database while continuing to receive and apply redo data from a primary database.

- There comes snapshot standby database which is new type of updatable standby database.

- You can use the RMAN DUPLICATE command to create a physical standby database over the network without a need for pre-existing database backups.

- Lost-write detection feature using a physical standby.

New Features in Oracle Data Guard 11.1 (Applicable to SQL Apply only)
- DBMS_RLS, DBMS_FGA PL/SQL Packages are supported and also XML stored as CLOB.

- Transparent Data Encryption (TDE) is supported in SQL apply. This allows a logical standby database to provide data protection for applications with advanced security requirements.

- Data Guard SQL Apply parameters can be set dynamically.

- When switching over to a logical standby database where either the primary database or the standby database is using Oracle RAC, the SWITCHOVER command can be used without having to shut down any instance, either at the primary or at the logical standby database.

- Scheduler Jobs can be created on a standby database using the PL/SQL DBMS_SCHEDULER package and can be associated with an appropriate database role so that they run when intended (for example, when the database is the primary, standby, or both).


New Features in Oracle Data Guard 11.2 (Applicable to both Redo Apply and SQL Apply)

- Data Guard configuration now support up to 30 standby databases.

- In Oracle 10g there was introduced flash recovery area. Starting from Oracle 11g the place is known as fast recovery area. In 11gR2 the fast recovery area location changed from LOG_ARCHIVE_DEST_10 to LOG_ARCHIVE_DEST_1.

- The initialization parameter FAL_CLIENT is deprecated.

- Redo transport compression is no longer limited to compressing redo data only when a redo gap is being resolved. When compression is enabled, all redo data sent is compressed.

- The ALTER SYSTEM FLUSH REDO SQL statement can be used at failover time to flush unsent redo from a mounted primary database to a standby database, thereby allowing a zero data loss failover even if the primary database is not running in a zero data loss data protection mode.


New Features in Oracle Data Guard 11.2 (Applicable to both Redo Apply only)

- Apply lag tolerance can be configured by using the new STANDBY_MAX_DATA_DELAY parameter.

- The "ALTER SESSION SYNC WITH PRIMARY" SQL statement can be used to ensure that a physical standby database is synchronized with the primary database as of the time the statement is issued.

- The V$DATAGUARD_STATS view now has apply lag and transport lag columns. Data inside the view columns also contain more accurate information.

- The new V$STANDBY_EVENT_HISTOGRAM view has been introduced where we can see the histogram of apply lag values on the physical standby database.

- A corrupted data block in a primary database can be automatically replaced with an uncorrupted copy of that block from a physical standby database and vice versa.

New Features in Oracle Data Guard 11.2 (Applicable to only SQL Apply)
- Logical standby databases and the LogMiner utility support tables with basic table compression and OLTP table compression.

- Logical standby and the LogMiner utility support tables with SecureFile LOB columns. Compression and encryption operations on SecureFile LOB columns are also supported.

- Online redefinition performed at the primary database using the DBMS_REDEFINITION PL/SQL package is transparently replicated on a logical standby database.

- Logical Standby supports the use of editions at the primary database, including the use of edition-based redefinition to upgrade applications with minimal downtime.

- Logical standby databases support Streams Capture. This allows you to offload processing from the primary database in one-way information propagation configurations and make the logical standby the hub that propagates information to multiple databases. Streams Capture can also propagate changes that are local to the logical standby database.

NTP and csstd time synchronization option to install Oracle Clusterware 11gR2

In Oracle 11gR2 and later version, while installing Oracle clusterware it is required time synchronization across all nodes within a cluster. We can use two different options to setup time synchronization.

1)Operating system configured Network Time Protocol (NTP).
2)Oracle Cluster Time Synchronization Service (ctssd).

If you don't have NTP services started then Oracle Cluster Time Synchronization Service is used. However if you want to use ctssd (Cluster Time Synchronization Service Daemon) service in the cluster, then deconfigure and deinstall the Network Time Protocol (NTP).

In order to deactivate NTP service, you need to follow the following procedures.

i)Stop the existing ntpd service.
# /sbin/service ntpd stop

ii)Disable the service from the initialization sequences so that it does not start during startup.
# chkconfig ntpd off

iii)Remove the ntp.conf file.
# rm /etc/ntp.conf
or,
# mv /etc/ntp.conf to /etc/ntp.conf.bak

Also remove the file /var/run/ntpd.pid which maintains the pid for the NTP daemon.
# rm /var/run/ntpd.pid

When the installer finds that the NTP protocol service is inactive, the Cluster Time Synchronization Service is installed in active mode and synchronizes the time across the nodes.

IF NTP is active on the server, then the Cluster Time Synchronization Service is started in observer mode, and no active time synchronization is performed by Oracle Clusterware within the cluster.

To check whether ctssd is active or not as grid installation owner just issue,
$ crsctl check ctss

If you want to continue using NTP time synchronization service, then you need to modify the NTP initialization file to set the -x flag, which prevents time from being adjusted backward. After you modify it restart ntpd. Following steps demonstrate the procedure on Oracle Enterprise Linux, Red Hat Linux.

i)Edit /etc/sysconfig/ntpd file and add -x flag.
# vi /etc/sysconfig/ntpd
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"
SYNC_HWCLOCK=no

ii)Restart NTP service.
# /sbin/service ntp restart

On SUSE systems, modify the configuration file /etc/sysconfig/ntp with the following settings.
NTPD_OPTIONS="-x -u ntp"

Related Documents
http://arjudba.blogspot.com/2010/03/cluvfy-fails-with-prvf-5436-prvf-9652.html
http://arjudba.blogspot.com/2010/03/in-11gr2-grid-rootsh-fails-with-crs.html
http://arjudba.blogspot.com/2010/03/what-to-do-after-failure-of-oracle.html
http://arjudba.blogspot.com/2009/12/enable-archive-log-mode-for-rac.html
http://arjudba.blogspot.com/2008/09/list-of-parameters-that-must-have.html
http://arjudba.blogspot.com/2008/08/oracle-rac-software-components.html
http://arjudba.blogspot.com/2008/08/oracle-clusterware-processes-on-unix.html
http://arjudba.blogspot.com/2008/08/configure-raw-devices-for-asm-in-rac.html
http://arjudba.blogspot.com/2008/08/crs-stack-fails-to-start-after-reboot.html
http://arjudba.blogspot.com/2008/08/configure-network-for-oracle-rac.html
http://arjudba.blogspot.com/2008/08/pre-installation-rac-environement-setup.html
http://arjudba.blogspot.com/2008/08/configure-server-to-install-oracle-rac.html

cluvfy fails with PRVF-5436 PRVF-9652 Cluster Time Synchronization Services check failed

Problem Description
"cluvfy comp clocksync" check fails with PRVF-5436 and PRVF-9652 error like below.

$GRID_HOME/bin/cluvfy comp clocksync
..
..
..
Checking daemon liveness...
Liveness check passed for "xntpd"
PRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing option "slewalways yes"
Clock synchronization check using Network Time Protocol(NTP) failed
PRVF-9652 : Cluster Time Synchronization Services check failed

From $ORACLE_BASE/oraInventory/logs/installActions{$TIMESTAMP}.log installation logs it shows similar entries like following:

..
INFO: Checking daemon liveness...
INFO: Liveness check passed for "xntpd"
INFO: PRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing option "slewalways yes"
INFO: Clock synchronization check using Network Time Protocol(NTP) failed
INFO: PRVF-9652 : Cluster Time Synchronization Services check failed
INFO: Post-check for cluster services setup was unsuccessful on all the nodes.

Cause of the Problem
The problem happened because the time service (such as NTP) is not running or doesn't have slew option set.

Solution of the Problem
The problem happened because time service NTP is not running or doesn't have slew option set.
So, ensure that NTP service is up and running.
Based on the operating system solutions are given below.

1) Linux:
To verify service as root user issue,
# /sbin/service ntpd status
ntpd (pid 4423) is running...

Check process is running or not by.
# ps -ef|grep ntp
ntp 4209 1 0 Mar10 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -x

# grep OPTIONS /etc/sysconfig/ntpd
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -x"

If NTP service is not started then to start the service issue,
To start the service issue,

# /sbin/service ntpd start

2) Solaris:
To verify the service issue,

# /usr/bin/svcs ntp
STATE STIME FMRI
online 3:29:11 svc:/network/ntp:default

# ps -ef|grep ntp
root 21223 1 0 Mar 10 ? 0:21 /usr/lib/inet/xntpd

# grep slewalways /etc/inet/ntp.conf
slewalways yes

To start the NTP service issue,

# /usr/sbin/svcadm enable ntp

3) HP-UX:

To verify the service issue,

# ps -ef|grep ntp
root 6022 1 0 14:23:42 ? 0:01 /usr/sbin/xntpd -x

# grep XNTPD_ARGS /etc/rc.config.d/netdaemons
export XNTPD_ARGS="-x"

To start the service issue,
# /sbin/init.d/xntpd start

4) AIX:

To verify the service issue,

# /usr/bin/lssrc -ls xntpd
xntpd tcpip 368754 active

# ps -ef|grep ntp
root 786614 151686 0 08:02:32 - 0:00 /usr/sbin/xntpd -x

# grep xntpd /etc/rc.tcpip
start /usr/sbin/xntpd "$src_running" -a "-x"

To start the service issue,

# /usr/bin/startsrc -s xntpd -a "-x"
Related Documents
http://arjudba.blogspot.com/2010/03/cluvfy-fails-with-prvf-5436-prvf-9652.html
http://arjudba.blogspot.com/2010/03/in-11gr2-grid-rootsh-fails-with-crs.html
http://arjudba.blogspot.com/2010/03/what-to-do-after-failure-of-oracle.html
http://arjudba.blogspot.com/2009/12/enable-archive-log-mode-for-rac.html
http://arjudba.blogspot.com/2008/09/list-of-parameters-that-must-have.html
http://arjudba.blogspot.com/2008/08/oracle-rac-software-components.html
http://arjudba.blogspot.com/2008/08/oracle-clusterware-processes-on-unix.html
http://arjudba.blogspot.com/2008/08/configure-raw-devices-for-asm-in-rac.html
http://arjudba.blogspot.com/2008/08/crs-stack-fails-to-start-after-reboot.html
http://arjudba.blogspot.com/2008/08/configure-network-for-oracle-rac.html
http://arjudba.blogspot.com/2008/08/pre-installation-rac-environement-setup.html
http://arjudba.blogspot.com/2008/08/configure-server-to-install-oracle-rac.html

In 11gR2 Grid root.sh fails with CRS-2674: Start of 'ora.cssd' failed

Problem Description
In Linux x86-64 system while installing Oracle Grid Infrastructure RAC cluster running $GRID_HOME/root.sh succeeds on the 1st node but fails on the 2nd node while attempting to start 'ora.cssd'.

Following is the log file entry from the cssd log.

CRS-2674: Start of 'ora.cssd' on 'rac2' failed
CRS-2679: Attempting to clean 'ora.cssd' on 'rac2'
CRS-2681: Clean of 'ora.cssd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'rac2'
CRS-2677: Stop of 'ora.diskmon' on 'rac2' succeeded
CRS-4000: Command Start failed, or completed with errors.
CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
CRS-2674: Start of 'ora.diskmon' on 'rac2' failed
CRS-2679: Attempting to clean 'ora.diskmon' on 'rac2'
CRS-5016: Process "/u01/oracle/11.2.0/grid/bin/diskmon" spawned by agent "/u01/oracle/11.2.0/grid/bin/orarootagent.bin" for action "clean" failed: details at "(:CLSN00010:)" in "/u01/oracle/11.2.0/grid/log/rac2/agent/ohasd/orarootagent_root/orarootagent_root.log"
CRS-2681: Clean of 'ora.diskmon' on 'rac2' succeeded
CRS-2674: Start of 'ora.cssd' on 'rac2' failed
CRS-2679: Attempting to clean 'ora.cssd' on 'rac2'
CRS-2681: Clean of 'ora.cssd' on 'rac2' succeeded
CRS-4000: Command Start failed, or completed with errors.
Command return code of 1 (256) from command: /u01/oracle/11.2.0/grid/bin/crsctl start resource ora.ctssd -init -env USR_ORA_ENV=CTSS_REBOOT=TRUE
Start of resource "ora.ctssd -init -env USR_ORA_ENV=CTSS_REBOOT=TRUE" failed
Failed to start CTSS
Failed to start Oracle Clusterware stack

Cause of the Problem
The startup of the CSS daemon on RAC node 2 failed because either it could not establish a network connection to the first node or it could not synchronize time with the first node. You can be sure by reviewing the messages seen in the CSS daemon log ($GRID_HOME/log/{nodename}/cssd/ocssd.log) on the 2nd node. If your 2nd node hostname is rac2 then review at location $GRID_HOME/log/rac2/cssd/ocssd.log.

From is the part of ocssd.log entry.

2010-03-13 10:59:36.581: [ CSSD][1246480704]clssnmLocalJoinEvent: Node rac1, number 1, is in an existing cluster with disk state 3
2010-03-13 10:59:36.582: [ CSSD][1246480704]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
2010-03-13 10:59:36.685: [ CSSD][1162561856]clssnmvDHBValidateNCopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 157026738, wrtcnt, 1507, LATS 66284084, lastSeqNo 1507, uniqueness 1261524838, timestamp 1261526376/66279084
2010-03-13 10:59:37.110: [ CSSD][1215011136]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2010-03-13 10:59:37.513: [ CSSD][1235990848]clssnmSendingThread: sending join msg to all nodes
2010-03-13 10:59:37.513: [ CSSD][1235990848]clssnmSendingThread: sent 5 join msgs to all nodes

While the connectivity on the cluster interconnect appeared to be functioning (nodes can be pinged via their private nodenames/IP addresses), a firewall blocked traffic on certain ports thus impacting the communication between the CRS daemon processes. It may also be reason that due to time synchronization problem running script root.sh on 2nd node fails.

Solution of the Problem
1) Disable Firewall:
Disable firewalls in all nodes. In Linux log in as root and run the following commands,
$ su
# service iptables stop
# service ip6tables stop

To disable firewall permanently issue,
# chkconfig iptables off
# chkconfig ip6tables off

If you want to enable firewall then exclude all traffic on the private network from the firewall configuration.

2) Synchronize Time between the nodes:
Setup NTP server to ensure that you have time synchronized between all nodes. In the post How to setup NTP Server it is discussed how to setup NTP server. So after synchronize timing deconfigure and reconfigure your grid infrastructure installation by following http://arjudba.blogspot.com/2010/03/what-to-do-after-failure-of-oracle.html.
Related Documents
http://arjudba.blogspot.com/2010/03/cluvfy-fails-with-prvf-5436-prvf-9652.html
http://arjudba.blogspot.com/2010/03/in-11gr2-grid-rootsh-fails-with-crs.html
http://arjudba.blogspot.com/2010/03/what-to-do-after-failure-of-oracle.html
http://arjudba.blogspot.com/2009/12/enable-archive-log-mode-for-rac.html
http://arjudba.blogspot.com/2008/09/list-of-parameters-that-must-have.html
http://arjudba.blogspot.com/2008/08/oracle-rac-software-components.html
http://arjudba.blogspot.com/2008/08/oracle-clusterware-processes-on-unix.html
http://arjudba.blogspot.com/2008/08/configure-raw-devices-for-asm-in-rac.html
http://arjudba.blogspot.com/2008/08/crs-stack-fails-to-start-after-reboot.html
http://arjudba.blogspot.com/2008/08/configure-network-for-oracle-rac.html
http://arjudba.blogspot.com/2008/08/pre-installation-rac-environement-setup.html
http://arjudba.blogspot.com/2008/08/configure-server-to-install-oracle-rac.html

What to do after failure of Oracle 11gR2 Grid Infrastructure (CRS) Installation

In this post I will discuss about how to proceed from failure of running root.sh script as well as failure before running of root.sh in Oracle 11gR2 Grid Infrastructure (CRS) Installation. Note that, the steps described in this post is only applicable for Oracle 11gR2 Grid Infrastructure installation.

I have discussed the things into four individual sections.
1) Deconfigure and Reconfigure of 11gR2 Grid Infrastructure installation in Standalone Server
2) Deinstall of 11gR2 Grid Infrastructure installation in Standalone Server
3) Deconfigure and Reconfigure of Grid Infrastructure Cluster
4) Deinstall of Grid Infrastructure Cluster

1) Deconfigure and Reconfigure of 11gR2 Grid Infrastructure installation in Standalone Server
To deconfigure,

i) Login as root and execute "$GRID_HOME/crs/install/roothas.pl -delete -force -verbose"
$ su
#$GRID_HOME/crs/install/roothas.pl -delete -force -verbose

ii) If it fails, please disable CRS, then reboot the node.

As root, execute "$GRID_HOME/bin/crsctl disable "
$su
# crsctl disable crs
# reboot
Once the node comes backup, execute above deconfigure command again.
# $GRID_HOME/crs/install/roothas.pl -delete -force -verbose

To reconfigure,
As root, execute "$GRID_HOME/root.sh"
$ su
# $GRID_HOME/root.sh

2) Deinstall of 11gR2 Grid Infrastructure installation in Standalone Server
As Grid user, execute $GRID_HOME/deinstall/deinstall
$ $GRID_HOME/deinstall/deinstall

3) Deconfigure and Reconfigure of Grid Infrastructure Cluster
Identify of root cause of failure of root.sh by reviewing the log files. Examine the logfiles from the location $GRID_HOME/cfgtoollogs/crsconfig and $GRID_HOME/log. Once cause is identified, follow the following steps one by one.

Step 01: As root, run "$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force" on all nodes, except the last one. If you have two nodes cluster then only run at first node.

$su
# $GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force

Step 02: As root, run "$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode" on last node. This command will zero out OCR and VD disk also. If you have two nodes cluster then run on second node.

# $GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode

Step 03: Fix the problem by identified cause in the log files.

Step 04: As root, run $GRID_HOME/root.sh on first node
# $GRID_HOME/root.sh

After completion run on all other nodes except last one. At last run at last node as root user.


4) Deinstall of Grid Infrastructure Cluster

Case 1: If "root.sh" is never run on the cluster, then as grid user, execute
$ $GRID_HOME/deinstall/deinstall

Case 2: If "root.sh" already ran, then follow the step below - please keep in mind that you will need wait till each step finishes successfully before move to next one.

Step 01: As root, run "$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force" on all nodes, except the last one.
$ su
# $GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force

Step 02: As root, run "$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode" on last node. This command will zero out OCR and VD disk also.
# $GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode

Step 03: As grid user, run $GRID_HOME/deinstall/deinstall
$ $GRID_HOME/deinstall/deinstall
Related Documents
http://arjudba.blogspot.com/2010/03/cluvfy-fails-with-prvf-5436-prvf-9652.html
http://arjudba.blogspot.com/2010/03/in-11gr2-grid-rootsh-fails-with-crs.html
http://arjudba.blogspot.com/2010/03/what-to-do-after-failure-of-oracle.html
http://arjudba.blogspot.com/2009/12/enable-archive-log-mode-for-rac.html
http://arjudba.blogspot.com/2008/09/list-of-parameters-that-must-have.html
http://arjudba.blogspot.com/2008/08/oracle-rac-software-components.html
http://arjudba.blogspot.com/2008/08/oracle-clusterware-processes-on-unix.html
http://arjudba.blogspot.com/2008/08/configure-raw-devices-for-asm-in-rac.html
http://arjudba.blogspot.com/2008/08/crs-stack-fails-to-start-after-reboot.html
http://arjudba.blogspot.com/2008/08/configure-network-for-oracle-rac.html
http://arjudba.blogspot.com/2008/08/pre-installation-rac-environement-setup.html
http://arjudba.blogspot.com/2008/08/configure-server-to-install-oracle-rac.html