Introduction:
ADOP cutover phase has failed in a PRODUCTION system while running a patch cycle. But the roles between Run and Patch file systems have switched, though the adop cutover has failed.
Before starting “adop cycle” the roles of the file systems were as shown below :
Run FS = FS2
Patch FS = FS1
After the failed “adop phase=cutover ” :
Run FS = FS1
Patch FS = FS2
Once we reach the cutover phase there is no going back and that too in our scenario file systems have also switched roles. If adop had failed during prepare or apply phase we could have aborted the adop cycle and run cleanup & fs_clone, in this case we cannot do that.
Errors:
Session killed.
Note that this is normal in the online patching cutover phase.
DB-Handle Error Code: 28
DB-Handle Error Message: ORA-00028: your session has been killed (DBD ERROR: OCIStmtExecute/Describe)
Reconnecting to the database. [ERROR] adop phase=cutover failed on Node: “ebsprodapp01”
[ERROR] adop phase=cutover failed on Node: “ebsprodapp02”
[ERROR] adop phase=cutover failed on Node: “ebsprodapp03”
[ERROR] adop phase=cutover failed on Node: “ebsprodapp04”
[UNEXPECTED]adop phase=cutover on all the nodes returned failure.
[UNEXPECTED]Unable to continue processing.
[UNEXPECTED]Error running “adop phase=cutover” on node(s): ebsprodapp01, ebsprodapp02, ebsprodapp03 and ebsprodapp04.
Summary report for current adop session:
Node ebsprodapp02: Failed
– Cutover status: FLIP_SNAPSHOTS_COMPLETED
Node ebsprodapp03: Failed
– Cutover status: FS_CUTOVER_COMPLETED
Node ebsprodapp04: Failed
– Cutover status: FS_CUTOVER_COMPLETED
Node ebsprodapp01: Failed
– Cutover status: FS_CUTOVER_COMPLETED
For more details, run the command: adop -status -detail
The log file of Node 4 has this failure message :
File System Cutover did not complete on node(s): <ebsprodapp01,ebsprodapp02,ebsprodapp03>
Will wait for another minute and check again.
[UNEXPECTED]Detected a failed session on another node(s): ebsprodapp02 and [UNEXPECTED]This session will also fail.
Error from adop.log in NODE 2:
[EVENT] Performing file system cutover.[PROCEDURE] [START 2024/03/16 21:45:57] Performing File System Cutover
[UNEXPECTED]Error occurred running “perl /u01_ebsprod_01/apps/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl -contextfile=/u01_ebsprod_01/apps/fs2/inst/apps/PRODEBS_ebsprodapp02/appl/admin/PRODEBS_ebsprodapp02.xml -patchcontextfile=/u01_ebsprod_01/apps/fs1/inst/apps/PRODEBS_ebsprodapp02/appl/admin/PRODEBS_ebsprodapp02.xml -promptmsg=hide -console=off -sessionid=191 -timestamp=20240316_213553 -outdir=/u01_ebsprod_01/apps/fs_ne/EBSapps/log/adop/191/20240316_213553/cutover/u01ap02 -action=ctxupdate”
[UNEXPECTED]Cutover phase has failed.
Analysis :
We have a four-node application tier setup and the cutover stopped at FLIP_SNAPSHOTS_COMPLETED on one of the Application Nodes (node2), but it moved to FS_CUTOVER_COMPLETED on the other nodes as shown below.
set pagesize 200;
set linesize 160;
col PREPARE_STATUS for a15
col APPLY_STATUS for a15
col CUTOVER_STATUS for a15
col CLEANUP_STATUS for a15;
col ABORT_STATUS for a15
col STATUS for a15
SQL> select NODE_NAME,ADOP_SESSION_ID, PREPARE_STATUS , APPLY_STATUS ,CUTOVER_STATUS , CLEANUP_STATUS , ABORT_STATUS , STATUS from AD_ADOP_SESSIONS where ADOP_SESSION_ID=191 order by 1;
NODE_NAME ADOP_SESS_ID PREP_STATUS APPLY_STATUS CUTOVER_STATUS CLNUP_STATUS ABORT_STATUS STATUS
——————— ————————- ———————– ——————- ———————— ———————- ———————— ———-
ebsprodapp01 191 Y Y 4 N X F
ebsprodapp02 191 Y Y D N X F
ebsprodapp03 191 Y Y 4 N X F
ebsprodapp04 191 Y Y 4 N X F
Please note the status of ebsprodapp02 : CUTOVER_STATUS=’D’ and rest of the application nodes have the status as ‘4’
Cutover status codes:
cutover_status=’Y”COMPLETED’
cutover_status not in (‘N’,’Y’,’X’) and status=’F”FAILED’
cutover_status=’0′ ‘CUTOVER STARTED’
cutover_status=’1′ ‘SERVICES SHUTDOWN COMPLETED’
cutover_status=’3′ ‘DB CUTOVER COMPLETED’
cutover_status=‘D’ ‘FLIP SNAPSHOTS COMPLETED’
cutover_status=‘4’ ‘FS CUTOVER COMPLETED’
cutover_status=’5′ ‘ADMIN STARTUP COMPLETED’
cutover_status=’6′ ‘SERVICES STARTUP COMPLETED’
cutover_status=’N’ ‘NOT STARTED’
cutover_status=’X’ ‘NOT APPLICABLE’
Conclusion/Solution :
Run “adop phase=cutover” again. Cutover will pick up the status of each node from the ADOP tables and run from point of failure.
Most of the times the solution looks simple, but the crux of the matter is to identify and understand the concept before executing the fix.
