R12_2 application patching adop cutover failed

Issue:  

EBS R12.2 application patching “adop cutover” got failed with following error.

Node Primary: Failed

Cutover status: FS_CUTOVER_COMPLETED

Node DMZ: Failed

Cutover status: FLIP_SNAPSHOTS_COMPLETED

adop exiting with status = 2 (Fail)

 

Cause of the issue:

A network drop  (or) glitch made Database connection to drop, which caused the above error during “cutover”.

Output from ad_adop_sessions

Primary 52 Y Y 4 N X F

DMZ 52 Y Y D N X F

EBS PRODUCTION application is down and will not be available to users until the issue is resolved. 

 

Solution:

Step 1: Run the below query which give current status of ADOP

select NODE_NAME,ADOP_SESSION_ID, PREPARE_STATUS , APPLY_STATUS  ,CUTOVER_STATUS , CLEANUP_STATUS , ABORT_STATUS , STATUS from AD_ADOP_SESSIONS order by ADOP_SESSION_ID;

 

Step 2: Run the below command to complete the adop patching cycle from Primary (or) Master node

adop phase=cutover workers=16 mtrestart=no

 

Step 3: Start the application

Once completed successfully, try to start the application services from the new run file system (Both Primary & DMZ)

 

Conclusion:

Whenever ADOP “cutover” got failed ,first we should find out the phase at which the cutover got failed and based on that we must proceed towards resolving it.

The following query gives the status at which the cutover got failed and also the relevant details.

select NODE_NAME,ADOP_SESSION_ID, PREPARE_STATUS , APPLY_STATUS  ,CUTOVER_STATUS , CLEANUP_STATUS , ABORT_STATUS , STATUS from AD_ADOP_SESSIONS order by ADOP_SESSION_ID;

Numerical values are only relevant for the cutover phase.

cutover_status=’Y’ ‘COMPLETED’

cutover_status not in (‘N’,’Y’,’X’) and status=’F’ ‘FAILED’

cutover_status=’0′ ‘CUTOVER STARTED’

cutover_status=’1′ ‘SERVICES SHUTDOWN COMPLETED’

cutover_status=’3′ ‘DB CUTOVER COMPLETED’

cutover_status=’D’ ‘FLIP SNAPSHOTS COMPLETED’

cutover_status=’4′ ‘FS CUTOVER COMPLETED’

cutover_status=’5′ ‘ADMIN STARTUP COMPLETED’

cutover_status=’6′ ‘SERVICES STARTUP COMPLETED’

cutover_status=’N’ ‘NOT STARTED’

cutover_status=’X’ ‘NOT APPLICABLE’