ORACLE

Friday, October 7, 2016

Oracle Failed to request gap sequence

Primary and standby DB : 11.2.0.3 on RHEL 6

Primary DB: PROD

Standby DB: STANDBY

We got an alert for one of our production standby database is out of sync

Standby Alert log :

FAL[client]: Failed to request gap sequence

GAP - thread 1 sequence 32651-32656

DBID 3477518933 branch 814361045

FAL[client]: All defined FAL servers have been attempted.

------------------------------------------------------------

Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization

parameter is defined to a value that's sufficiently large

enough to maintain adequate log switch information to resolve

archivelog gaps.

------------------------------------------------------------

Fri Sep 23 15:43:48 2016

db_recovery_file_dest_size of 25600 MB is 0.00% used. This is a

user-specified limit on the amount of space that will be used by this

database for recovery-related files, and does not reflect the amount of

space available in the underlying filesystem or ASM diskgroup.

Fri Sep 23 15:54:41 2016

Then checked for the archive gap….

standby_server:STANDBY:/opt/oracle/ARCH/standby $ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Fri Sep 23 15:29:26 2016

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.Sfference" FROM (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROAD#)) ARCH, (SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HIS WHERE ARCH.THREAD# = APPL.THREAD# ORDER BY 1;

Thread Last Sequence Received Last Sequence Applied Difference

---------- ---------------------- --------------------- ----------

1 32687 32650 37 <<----- 37 arch logs gap

SQL> select message, timestamp from v$dataguard_status where severity in ('Error','Fatal')

order by timestamp;

no rows selected

SQL> SELECT PROCESS, STATUS FROM V$MANAGED_STANDBY;

PROCESS STATUS

--------- ------------

ARCH CONNECTED

MR(fg) WAIT_FOR_GAP --->> Recovery says WAITING FOR GAP

RFS IDLE

NOTE : Before going for any type of troubleshooting for standby databases first thing we need to check is where standby have required archivelogs or not.

standby_server:STANDBY:/opt/oracle/ARCH/standby $ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Fri Sep 23 16:14:25 2016

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> archive log list

Database log mode Archive Mode

Automatic archival Enabled

Archive destination /opt/oracle/ARCH/standby/

Oldest online log sequence 32685

Next log sequence to archive 0

Current log sequence 32690

SQL> SELECT * FROM V$ARCHIVE_GAP;

THREAD# LOW_SEQUENCE# HIGH_SEQUENCE#

---------- ------------- --------------

1 32651 32656

Check last sequence applied:

SQL> SELECT thread#, Max(sequence#) "Last Standby Seq Applied" FROM v$archived_log WHERE applied = 'YES' GROUP BY thread# ORDER BY 1;

THREAD# Last Standby Seq Applied

---------- ------------------------------

1 32650

$ cd /opt/oracle/ARCH/standby

standby_server:STANDBY:/opt/oracle/ARCH/standby $ ls -lrth

-rw-r----- 1 oracle dba 2.4M Sep 21 19:11 TS0001_0000032647_STANDBY0814361045.ARC

-rw-r----- 1 oracle dba 29M Sep 21 23:22 TS0001_0000032648_STANDBY0814361045.ARC

-rw-r----- 1 oracle dba 15K Sep 21 23:22 TS0001_0000032649_STANDBY0814361045.ARC

-rw-r----- 1 oracle dba 78K Sep 21 23:29 TS0001_0000032650_STANDBY0814361045.ARC

-rw-r----- 1 oracle dba 38M Sep 23 12:51 TS0001_0000032657_STANDBY0814361045.ARC

-rw-r----- 1 oracle dba 39M Sep 23 12:51 TS0001_0000032658_STANDBY0814361045.ARC

-rw-r----- 1 oracle dba 66M Sep 23 12:52 TS0001_0000032659_STANDBY0814361045.ARC

-rw-r----- 1 oracle dba 52M Sep 23 12:52 TS0001_0000032660_STANDBY0814361045.ARC

-rw-r----- 1 oracle dba 34M Sep 23 12:52 TS0001_0000032661_STANDBY0814361045.ARC

-rw-r----- 1 oracle dba 38M Sep 23 12:52 TS0001_0000032662_STANDBY0814361045.ARC

-rw-r----- 1 oracle dba 66M Sep 23 12:53 TS0001_0000032663_STANDBY0814361045.ARC

-rw-r----- 1 oracle dba 85M Sep 23 12:53 TS0001_0000032664_STANDBY0814361045.ARC

-rw-r----- 1 oracle dba 29M Sep 23 12:53 TS0001_0000032665_STANDBY0814361045.ARC

From the above output we can clearly see that from Seq# 32651 - seq# 32656 are MISSING, uummmm now we know what need to be done to resolve

NOTE: Luckily i have archivelogs exists in primary side if doesn't exists any more then you need to rebuild standby again(ex: rman incremental backups,etc.,)

Primary:

oracleprimaryserver:PROD:/opt/oracle/ARCH/Primary $ sftp oracle@standby_server:/opt/oracle/ARCH/standby

Connecting to standby_server...

oracle@standby_server's password:

Changing to: /opt/oracle/ARCH/standby

sftp> put TS0001_0000032651_STANDBY0814361045.ARC

Uploading TS0001_0000032651_STANDBY0814361045.ARC to /opt/oracle/ARCH/standby/TS0001_0000032651_STANDBY0814361045.ARC

TS0001_0000032651_STANDBY0814361045.ARC 100% 278KB 278.1KB/s 00:0

Similarly, all……………….

Once copied then STOP Recover process on standby:

Note: open another Terminal and watch the alert log

standby_server:STANDBY:/opt/oracle/ARCH/standby $ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Fri Sep 23 16:09:08 2016

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> Recover managed standby database cancel;

SQL> alter database register logfile '/FULL_PATH/TS0001_0000032651_STANDBY0814361045.ARC';

Database altered.

Similarly register all the 6 logfiles and start the recovery process

SQL> recover managed standby database;

Now check again the gap

SQL> SELECT PROCESS, STATUS FROM V$MANAGED_STANDBY;

PROCESS STATUS

--------- ------------

ARCH CONNECTED

MR(fg) APPLYING_LOG <<-- See recovery started applying

RFS IDLE

SQL> SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference"

FROM (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH,

(SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL

WHERE ARCH.THREAD# = APPL.THREAD# ORDER BY 1;

Thread Last Sequence Received Last Sequence Applied Difference

---------- ---------------------- --------------------- ----------

1 32689 32689 0

Hope this helps ….!!

Wednesday, August 3, 2016

thank you

Saturday, September 5, 2015

OGG-01496 Failed to open target trail file

My pump process is down for so many days and when I tried to start the process it is getting ABENDED because it is trying to write in the remote trail file which is MISSING

GGSCI (oracleqa011.domain.com) 7> start PUMPQA

Sending START request to MANAGER ...
EXTRACT PUMPQA starting

GGSCI (oracleqa011.domain.com) 8> info PUMPQA

EXTRACT PUMPQA Last Started 2015-08-20 19:59 Status ABENDED
Checkpoint Lag 00:00:00 (updated 00:00:28 ago)
Log Read Checkpoint File ./dirdat/cb000011 --> This is source trail file
2015-08-20 19:59:30.000000 RBA 36679

GGSCI (oracleqa011.domain.com) 9> view report PUMPQA

2015-08-20 19:59:44 ERROR OGG-01496 Failed to open target trail file /u01/oracle/TARGET/cb000008, at RBA 8448365.

2015-08-20 19:59:44 ERROR OGG-01668 PROCESS ABENDING.

Here pump process is trying to write to the file “cb000008” where is has left before, since it’s been long ago this file doesn’t exist in the target location

Tried doing begin now command, but doesn’t work

GGSCI (oracleqa011.domain.com) 10> alter PUMPQA begin now

EXTRACT altered.

Still “ABENDED” --> because it will begin now but try to write to the same trail file

Solution:

Do “ETROLLOVER” for pump process, this will roll over to the next trail file in the sequence (here cb000009)

GGSCI (oracleqa011.domain.com) 10> ALTER EXTRACT PUMPQA ETROLLOVER

2015-08-20 20:02:22 INFO OGG-01520 Rollover performed. For each affected output trail of Version 10 or higher format, after starting the source extract, issue ALTER EXTSEQNO for that trail's reader (either pump EXTRACT or REPLICAT) to move the reader's scan to the new trail file; it will not happen automatically.

EXTRACT altered.

GGSCI (oracleqa011.domain.com) 11> start PUMPQA

Sending START request to MANAGER ...

EXTRACT PUMPQA starting

GGSCI (oracleqa011.domain.com) 12> info PUMPQA

EXTRACT PUMPQA Last Started 2015-08-20 20:02 Status RUNNING

Checkpoint Lag 00:03:09 (updated 00:00:08 ago)

Log Read Checkpoint File ./dirdat/cb000011

2015-08-20 19:59:30.000000 RBA 36679

In the target server I see “cb000009” has been created and writing into it.

ERROR OGG-01224 Address already in use.

DB version: 11.2.0.4 2-Node RAC

OS: RHEL 6

GG version : 11.2.1.0.3

Today when i'm trying to start my manager process it is not starting

GGSCI (oracledev01) 1> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER STOPPED

JAGENT STOPPED

EXTRACT ABENDED EOECBP 291:56:23 27:02:50

GGSCI (oracledev01) 2> start mgr

Manager started.

But when I do info mgr it says manager is DOWN…..

GGSCI (oracledev01) 3> info mgr

Manager is DOWN!

GGSCI (oracledev01) 4> view params mgr

PORT 7809

DYNAMICPORTLIST 7840-9860

GGSCI (oracledev01) 5> view report mgr

***********************************************************************

Oracle GoldenGate Manager for Oracle

Version 11.2.1.0.14 17547423 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_131022.0605

Linux, x64, 64bit (optimized), Oracle 11g on Oct 22 2013 07:38:49

Starting at 2015-08-11 19:56:22

***********************************************************************

Operating System Version:

Linux

Version #1 SMP Fri May 29 10:16:43 EDT 2015, Release 2.6.32-504.23.4.el6.x86_64

Node: sl73orcdbdbq005

Machine: x86_64

soft limit hard limit

Address Space Size : unlimited unlimited

Heap Size : unlimited unlimited

File Size : unlimited unlimited

CPU Time : unlimited unlimited

Process id: 36788

Parameters...

PORT 7809

DYNAMICPORTLIST 7840-9860

***********************************************************************

** Run Time Messages **

***********************************************************************

Source Context :

SourceModule : [mgr.main]

SourceID : [/scratch/aime1/adestore/views/aime1_adc4150267/oggcore/OpenSys/src/app/mgr/mgr.c]

SourceFunction : [init_functions]

SourceLine : [3390]

ThreadBacktrace : [8] elements

: [/gg/GG11/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7ff4595509fe]]

: [/gg/GG11/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x2cc) [0x7ff45954974c]]

: [/gg/GG11/libgglog.so(_MSG_ERR_TCP_GENERIC(CSourceContext*, char const*, CMessageFactory::MessageDisposition)+0x31) [0x7ff4595318a5]]

: [./mgr(init_functions(int, char**)+0x7f5) [0x4511c5]]

: [./mgr(main_loop(int, char**)+0x4c) [0x454aec]]

: [./mgr(main+0xf2) [0x455362]]

: [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3d9361ed5d]]

: [./mgr(__gxx_personality_v0+0x142) [0x43efca]]

2015-08-11 19:56:22 ERROR OGG-01224 Address already in use.

2015-08-11 19:56:22 ERROR OGG-01668 PROCESS ABENDING.

Error in ggserr.log:

2015-08-11 19:56:10 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (oracle): info mgr.

2015-08-11 19:56:19 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (oracle): info all.

2015-08-11 19:56:22 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (oracle): start mgr.

2015-08-11 19:56:22 ERROR OGG-01224 Oracle GoldenGate Manager for Oracle, mgr.prm: Address already in use.

2015-08-11 19:56:22 ERROR OGG-01668 Oracle GoldenGate Manager for Oracle, mgr.prm: PROCESS ABENDING.

CAUSE:

Previous mgr process is still running, which used port 7809 according to parameter file.

SOLUTION:

Use other port and start

To release the port perform below steps

As a root or Goldengate owner check the port

[root@oracledev01~]# netstat -nap | grep 7809

tcp 0 0 0.0.0.0:7809 0.0.0.0:* LISTEN 44402/./mgr

From the above output we see mgr is already running on port 7809 so kill this old process and start again

[root@oracledev01~]# kill -9 44402

[root@oracledev01~]# netstat -nap | grep 7809

Nothing displays

[oracle@sl73orcdbdbq005 GG11]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle

Version 11.2.1.0.14 17547423 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_131022.0605_FBO

Linux, x64, 64bit (optimized), Oracle 11g on Oct 22 2013 11:03:39

GGSCI (oracledev01) 1> start mgr

Manager started.

GGSCI (oracledev01) 2> info mgr

Manager is running (IP port oracledev01.7809).

Thursday, August 13, 2015

ENABLE automatic statistics collection in 11g.

SQL> SHO PARAMETER STATISTICS_LEVEL

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

statistics_level string TYPICAL

SQL> select client_name, status,attributes,service_name from dba_autotask_client;

CLIENT_NAME STATUS ATTRIBUTES SERVICE_NAME

---------------------------------------- -------- ------------------------------------------------------------ ----------------------------

auto optimizer stats collection DISABLED ON BY DEFAULT, VOLATILE, SAFE TO KILL

auto space advisor ENABLED ON BY DEFAULT, VOLATILE, SAFE TO KILL

sql tuning advisor ENABLED ONCE PER WINDOW, ON BY DEFAULT, VOLATILE, SAFE TO KILL

SQL> select client_name,status from DBA_AUTOTASK_TASK;

CLIENT_NAME STATUS

---------------------------------------- --------

sql tuning advisor ENABLED

auto space advisor ENABLED

SQL> SELECT window_name,TO_CHAR(window_next_time,'DD-MON-YY HH24:MI:SS'),sql_tune_advisor, optimizer_stats, segment_advisor FROM DBA_AUTOTASK_WINDOW_CLIENTS;

WINDOW_NAME TO_CHAR(WINDOW_NEXT_TIME,'D SQL_TUNE OPTIMIZE SEGMENT_

------------------------------ --------------------------- -------- -------- --------

MONDAY_WINDOW 17-AUG-15 22:00:00 ENABLED DISABLED ENABLED

TUESDAY_WINDOW 18-AUG-15 22:00:00 ENABLED DISABLED ENABLED

WEDNESDAY_WINDOW 19-AUG-15 22:00:00 ENABLED DISABLED ENABLED

THURSDAY_WINDOW 13-AUG-15 22:00:00 ENABLED DISABLED ENABLED

FRIDAY_WINDOW 14-AUG-15 22:00:00 ENABLED DISABLED ENABLED

SATURDAY_WINDOW 15-AUG-15 06:00:00 ENABLED DISABLED ENABLED

SUNDAY_WINDOW 16-AUG-15 06:00:00 ENABLED DISABLED ENABLED

7 rows selected.

SQL>

SQL> SELECT ENABLED FROM DBA_SCHEDULER_PROGRAMS WHERE PROGRAM_NAME = 'GATHER_STATS_PROG';

ENABL

-----

TRUE

Check if this table has any pending stats

SQL> select LAST_ANALYZED,NUM_ROWS from dba_TAB_PENDING_STATS where TABLE_NAME='TEST_TAB';

no rows selected

Enable optimizer stats :

SQL> BEGIN

DBMS_AUTO_TASK_ADMIN.ENABLE(

client_name => 'auto optimizer stats collection',

operation => NULL,

window_name => NULL);

END;

PL/SQL procedure successfully completed.

SQL> SELECT window_name,TO_CHAR(window_next_time,'DD-MON-YY HH24:MI:SS'),sql_tune_advisor, optimizer_stats, segment_advisor FROM DBA_AUTOTASK_WINDOW_CLIENTS;

WINDOW_NAME TO_CHAR(WINDOW_NEXT_TIME,'D SQL_TUNE OPTIMIZE SEGMENT_

------------------------------ --------------------------- -------- -------- --------

MONDAY_WINDOW 17-AUG-15 22:00:00 ENABLED ENABLED ENABLED

TUESDAY_WINDOW 18-AUG-15 22:00:00 ENABLED ENABLED ENABLED

WEDNESDAY_WINDOW 19-AUG-15 22:00:00 ENABLED ENABLED ENABLED

THURSDAY_WINDOW 13-AUG-15 22:00:00 ENABLED ENABLED ENABLED

FRIDAY_WINDOW 14-AUG-15 22:00:00 ENABLED ENABLED ENABLED

SATURDAY_WINDOW 15-AUG-15 06:00:00 ENABLED ENABLED ENABLED

SUNDAY_WINDOW 16-AUG-15 06:00:00 ENABLED ENABLED ENABLED

7 rows selected.

Automatic Statistics Collection (Doc ID 1233203.1)

OGG-01224 Oracle GoldenGate Manager for Oracle, mgr.prm: Address already in use.

DB version: 11.2.0.4 2-Node RAC

OS: RHEL 6

GG version : 11.2.1.0.3

GGSCI (oracledev01) 1> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER STOPPED

JAGENT STOPPED

EXTRACT ABENDED EOECBP 291:56:23 27:02:50

GGSCI (oracledev01) 2> start mgr

Manager started.

But when I do info mgr it says manager is DOWN…..

GGSCI (oracledev01) 3> info mgr

Manager is DOWN!

GGSCI (oracledev01) 4> view params mgr

PORT 7809

DYNAMICPORTLIST 7840-9860

GGSCI (oracledev01) 5> view report mgr

***********************************************************************

Oracle GoldenGate Manager for Oracle

Version 11.2.1.0.14 17547423 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_131022.0605

Linux, x64, 64bit (optimized), Oracle 11g on Oct 22 2013 07:38:49

Starting at 2015-08-11 19:56:22

***********************************************************************

Operating System Version:

Linux

Version #1 SMP Fri May 29 10:16:43 EDT 2015, Release 2.6.32-504.23.4.el6.x86_64

Node: sl73orcdbdbq005

Machine: x86_64

soft limit hard limit

Address Space Size : unlimited unlimited

Heap Size : unlimited unlimited

File Size : unlimited unlimited

CPU Time : unlimited unlimited

Process id: 36788

Parameters...

PORT 7809

DYNAMICPORTLIST 7840-9860

***********************************************************************

** Run Time Messages **

***********************************************************************

Source Context :

SourceModule : [mgr.main]

SourceID : [/scratch/aime1/adestore/views/aime1_adc4150267/oggcore/OpenSys/src/app/mgr/mgr.c]

SourceFunction : [init_functions]

SourceLine : [3390]

ThreadBacktrace : [8] elements

: [/gg/GG11/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7ff4595509fe]]

: [/gg/GG11/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x2cc) [0x7ff45954974c]]

: [/gg/GG11/libgglog.so(_MSG_ERR_TCP_GENERIC(CSourceContext*, char const*, CMessageFactory::MessageDisposition)+0x31) [0x7ff4595318a5]]

: [./mgr(init_functions(int, char**)+0x7f5) [0x4511c5]]

: [./mgr(main_loop(int, char**)+0x4c) [0x454aec]]

: [./mgr(main+0xf2) [0x455362]]

: [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3d9361ed5d]]

: [./mgr(__gxx_personality_v0+0x142) [0x43efca]]

2015-08-11 19:56:22 ERROR OGG-01224 Address already in use.

2015-08-11 19:56:22 ERROR OGG-01668 PROCESS ABENDING.

Error in ggserr.log:

2015-08-11 19:56:10 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (oracle): info mgr.

2015-08-11 19:56:19 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (oracle): info all.

2015-08-11 19:56:22 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (oracle): start mgr.

2015-08-11 19:56:22 ERROR OGG-01224 Oracle GoldenGate Manager for Oracle, mgr.prm: Address already in use.

2015-08-11 19:56:22 ERROR OGG-01668 Oracle GoldenGate Manager for Oracle, mgr.prm: PROCESS ABENDING.

CAUSE:

Previous mgr process is still running, which used port 7809 according to parameter file.

SOLUTION:

Use other port and start

To release the port perform below steps

As a root or Goldengate owner check the port

[root@oracledev01~]# netstat -nap | grep 7809

tcp 0 0 0.0.0.0:7809 0.0.0.0:* LISTEN 44402/./mgr

From the above output we see mgr is running on port 7809 so kill this old process and start again

[root@oracledev01~]# kill -9 44402

[root@oracledev01~]# netstat -nap | grep 7809

[oracle@sl73orcdbdbq005 GG11]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle

Version 11.2.1.0.14 17547423 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_131022.0605_FBO

Linux, x64, 64bit (optimized), Oracle 11g on Oct 22 2013 11:03:39

GGSCI (oracledev01) 1> start mgr

Manager started.

GGSCI (oracledev01) 2> info mgr

Manager is running (IP port oracledev01.7809).

Wednesday, August 5, 2015

ORA-04045: errors during recompilation/revalidation of GG_USER.DDLREPLICATION

DB version: 11.2.0.4

OS: RHEL 6

GG version : 11.2.1.0.3

For any DDL operation performed in the database I’m hitting the below error

EX:

SQL> ALTER TABLE USER.TEST MODIFY (emp_id VARCHAR2(100) );

ALTER TABLE USER.TEST MODIFY (emp_id VARCHAR2(100) )

ERROR at line 1:

ORA-00604: error occurred at recursive SQL level 1

ORA-04045: errors during recompilation/revalidation of GGS_ADMIN.DDLREPLICATION

ORA-04067: not executed, package body "GGS_ADMIN.DDLREPLICATION" does not exist