15 Replies Latest reply: Mar 9, 2017 4:01 PM by zhaos2 RSS

Config to make recoverpoint work with Oracle 12c and ASM?

Quinc

Trying to setup EMC Recoverpoint lun to lun replication with Oracle 12c RAC 2 node cluster to Disaster Recovery env RAC one node cluster using ASM

 

We get Oracle up and working in the single node DR env and turn on recoverypoint.  Once the replication is completed we turn the one node cluster back to read/write and when we try to sqlplus Oracle gives us the error:

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4530: Communications failure contacting Cluster Synchronization Services daemon

CRS-4534: Cannot communicate with Event Manager


We reinstalled Oracle and the same thing happens once the luns have replicated.


Is there anything special with 12c and ASM that needs to be done for recoverpoint or any specific config with Oracle that needs to be done?


-Recoverpoint 4.0.SP2(m.24)

-RHEL 6.8

-Oracle 12c
-This was working previously with Oracle 11g


Any help would be much appreciated!

  • 1. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    Allanr

    From Oracle Database 11g R2 on , Oracle ASM was extended to include support for OCR and voting files to be placed within ASM disk groups.

     

     

    Your ASM Disk groups will look something like

     

     

    +OCR - OCR and voting files for cluster

     

     

    +DATA - Datafiles, controlfiles

    +REDO - current redo logs, controlfiles

    +FRA - archive redo logs, controlfiles

     

     

    Only +DATA, +REDO and +FRA should be replicated.

    The +OCR should be on separate luns and should not be replicated as these hold cluster specific info on ASM, subnet and host.

     

     

    On the DR site, have you  installed the Oracle GI and Database software and configured a single node RAC cluster.

    Are the cluster services started  on the new node?

  • 2. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    Quinc

    @ On the DR site we have installed the Grid infrastructure and database software for single node RAC cluster and the cluster services where started. We confirmed that we could import a schema and connect to it with our application and sqlplus etc.  Then after turning on replication, then turning off replication the next day after it was 100% replicated; it breaks Oracle.

     

    We only have one LUN which contains all DATA REDO FRA and OCR

     

    Is there a specific way in Recoverpoint that we need to turn off replication before we can log into Oracle? Or any thing we need to do with ASM before we can log into Oracle?

     

    Edit:  Instead of rebuilding Oracle again on the DR site can I use recoverpoint to take a snapshot of the prod environment and bring it over?

    Also we are going to remove replication on a 20GB lun and move the DATA REDO FRA and OCR etc onto it.

     

    Thank you for help! =)

  • 3. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    Quinc

    Data has been replicated and we are ready to test again.

     

    Before we stop replication:

    -Is there anything we need to do with Oracle? 

    -Also is there anything we need to do after we stop replication?

  • 4. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    Quinc

    We where able to bring up CRS this time but the data disks are not showing up now. Does anyone know why I can no longer see the data disks? Do I need to reboot the system or run anything?

    emc_data.PNG.png

    tenor.gif

  • 5. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    Quinc

    ./srvctl start db -d demoprd

    PRCR-1079 : Failed to start resource ora.demoprd.db

    CRS-2674: Start of 'ora.DATA.dg' on 'server1' failed

    CRS-5017: The resource action "ora.FRA.dg start" encountered the following error:

    ORA-15032: not all alterations performed

    ORA-15017: diskgroup "FRA" cannot be mounted

    ORA-15040: diskgroup is incomplete

    ORA-15080: synchronous I/O operation failed to read block 0 of disk 0 in disk group

    ORA-15080: synchronous I/O operation failed to read block 0 of disk 1 in disk group

    ORA-15186: ASMLIB error function = [kfk_asm_ioerror],  error = [0],  mesg = [I/O Error]

    ORA-15186: ASMLIB error function = [kfk_asm_ioerror],  error = [0],  mesg = [I/O Error]

     

     

    Are there any scripts that need to be run after turning off replication?

  • 6. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    zhaos2

    Hi Quinc, sorry for the late reply as I was in 2 weeks leave.

     

    As Allan instructed:

     

    Only +DATA, +REDO and +FRA should be replicated.

    The +OCR should be on separate luns and should not be replicated as these hold cluster specific info on ASM, subnet and host.

     

    Can you explain:

    'Also we are going to remove replication on a 20GB lun and move the DATA REDO FRA and OCR etc onto it.'

     

    Did you separated OCR from DATA/REDO/FRA, and replicate only 'DATA/REDO/FRA', and configure the OCR at the DR site?  If you turn of replication on the 20GB LUN with all DATA/REDO/FRA and OCR on it, of course you will not see DATA/REDO/FRA at the DR site.

  • 7. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    Quinc

    Thank you for the reply Zhaos!

     

    Only DATA REDO and FRA are being replicated. However, when we turn replication off to test we are no longer able to see the DATA REDO and FRA disks.  Is there something else we need to do after we turn off replication so we can see these disks? Like a command, or script we need to run to be able to see/connect Oracle to those disks?

  • 8. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    zhaos2

    Please refer to the white paper: 

    https://support.emc.com/docu7899_Replicating-Oracle-with-EMC-RecoverPoint-Technical-Notes.pdf?language=en_US  

     

    It is a bit old but detailed on how to replicate Oracle database with RP.

     

    Not sure how you turn off the replication, you may need to perform 'First Time Failover' operation  after replication as described in the paper.

     

    If it is for a Dell EMC customer, I would suggest to log a support case, and tell me the case number, I will try to work with the RP support engineer for you.

  • 9. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    Quinc

    Hey Zhaos I really appreciate the help! We have talked to multiple EMC support techs and no one seems to know if RP works with Oracle 12c and ASM.

     

    To test we have been turning off replication and putting RP into Test Copy. 

     

    Is this what we need to do for first time testing even if we are stopping replication?

     

     

    Use RecoverPoint image access to test a bookmarked snapshot taken while

    the Oracle database is in hot backup mode. Use the following procedure to test a snapshot:

    1.

    Choose a

    hot backup

    bookmark to access. Use the RecoverPoint Management

    Application or the CLI to enable image access. If you are using the CLI, run the following

    command:

    enable_image_access group=

    <consistency_group> copy=<copy name> image=<image name>

    Note

    Refer to Table 2 on page 7 to enable image access to the required hot backup

    consistency groups and to make sure that the archived redo consistency group

    book

    mark was created after the log switch.

    2.

    If not using raw disks, mount the file system, including each relevant volume at its

    mount point. For Solaris, use the mount command or for ASM use the ASM mount

    command.

    3.

    Use the

    following SQL commands to open the Or

    acle database at the

    replica

    site

    :

    1.

    Mount the database:

    Use the Oracle parameter file specifying the backup control file (init_bctrl.ora):

    startup mount pfile=<init_bctrl.ora>

    2.

    Perform media recovery using backup controlfile generated during bookmark

    creation:

    recover automatic database using backup controlfile until cancel

    3.

    Open the database:

    alter database open RESETLOGS

    For example:

    Temp-

    Sun-4% sqlplus "/ as sysdba"

    SQL*Plus: Release 11.2.0.1.0 Production on Tue May 4 14:01:09 2010

    Copyright (c) 1982, 20

    09, Oracle. All rights reserved.

    Connected to an idle instance.

    10

    Replication with RecoverPoint

    EMC

    ®

    RecoverPoint Replicating Oracle with RecoverPoint Technical Notes

    SQL>

    startup mount pfile=initRPDB.ora

    ORACLE instance started.

    Total System Global Area 217219072 bytes

    Fixed Size 2146664 bytes

    Variable Size 159391384 bytes

    Database Buffers 50331648 bytes

    Redo Buffers 5349376 bytes

    Database mounted.

    SQL>

    recover automatic database using backup controlfile until cancel

    ORA-00279: change 475319 generated at 05/04/2010 12:48:17 needed for

    thread 1

    ORA-00289: suggestion: +ARCH_DG

    ORA-00280: change 475319 for thread 1 is in sequence #9

    ORA-00278: log file '+ARCH_DG' no longer needed for this recovery

    ORA-00308: cannot open archived log '+ARCH_DG'

    ORA-17503: ksfdopn:2 Failed to open file +ARCH_DG

    ORA-15045: ASM file name '+ARCH_DG' is not in reference form

    Specify log: {<RET>=suggested | filename | AUTO | CANCEL}

    CANCEL

    Media recovery cancelled.

    SQL>

    alter database open RESETLOGS;

  • 10. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    Quinc

    I am not sure how to message you the SR number?

  • 12. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    Quinc

    Still no luck, looks like the Oracle VOL are being removed and then not being replicated over.

     

    Doing some more reading on multipath and asm, do I need to run pvcreate /dev/mapper/mpath so the disks are seen as a physica device?

     

    From redhat:

    After creating multipath devices, you can use the multipath device names just as you would use a physical device name when creating an LVM physical volume. For example, if /dev/mapper/mpatha is the name of a multipath device, the following command will mark /dev/mapper/mpatha as a physical volume.

    pvcreate /dev/mapper/mpatha 
  • 13. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    zhaos2

    Can we talk by email, are you Cody Kratz?

  • 14. Re: Config to make recoverpoint work with Oracle 12c and ASM?
    zhaos2

    Enabling disk UUID on virtual machines

    You must set the disk.EnableUUID parameter for each VM to "TRUE". This step is necessary so that the VMDK always presents a consistent UUID to the VM, thus allowing the disk to be mounted properly. For each of the virtual machine nodes (VMs) that will be participating in the cluster, follow the steps below from the vSphere client:

    To enable disk UUID on a virtual machine

    1. Power off the guest.
    2. Select the guest and select Edit Settings.
    3. Select the Options tab on top.
    4. Select General under the Advanced section.
    5. Select the Configuration Parameters... on right hand side.
    6. Check to see if the parameter disk.EnableUUID is set, if it is there then make sure it is set to TRUE.If the parameter is not there, select Add Row and add it.
    7. Power on the guest.