ORA07445 Exception [Address not mapped to object][ kglrfcl()+492]

客户的环境数据库版本oracle12.2.0.1，redhat7.9，单机模式。数据库出现宕机，手动重启后恢复正常。记录如下：

错误日志，查看alert日志信息。

2023-06-11T22:15:10.128319+08:00
Archived Log entry 66582 added for T-1.S-66573 ID 0x8bb76fc1 LAD:1
2023-06-11T22:15:11.809567+08:00
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x8] [PC:0x10815F0C, kglrfcl()+492] [flags: 0x0, count: 1]
Errors in file /data1/oradata/diag/rdbms/oradb/oradb/trace/oradb_ora_18036.trc (incident=240700):
ORA-07445: 出现异常错误: 核心转储 [kglrfcl()+492] [SIGSEGV] [ADDR:0x8] [PC:0x10815F0C] [Address not mapped to object] []
Incident details in: /data1/oradata/diag/rdbms/oradb/oradb/incident/incdir_240700/oradb_ora_18036_i240700.trc
Dumping diagnostic data in directory=[cdmp_20230611221512], requested by (instance=1, osid=18036), summary=[incident=240700].
2023-06-11T22:15:18.386955+08:00
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x8] [PC:0x10815F0C, kglrfcl()+492] [flags: 0x0, count: 1]
Errors in file /data1/oradata/diag/rdbms/oradb/oradb/trace/oradb_clmn_7090.trc  (incident=240028):
ORA-07445: 出现异常错误: 核心转储 [kglrfcl()+492] [SIGSEGV] [ADDR:0x8] [PC:0x10815F0C] [Address not mapped to object] []
Incident details in: /data1/oradata/diag/rdbms/oradb/oradb/incident/incdir_240028/oradb_clmn_7090_i240028.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2023-06-11T22:15:19.270397+08:00
Errors in file /data1/oradata/diag/rdbms/oradb/oradb/trace/oradb_clmn_7090.trc:
ORA-00602: 内部编程异常错误
ORA-07445: 出现异常错误: 核心转储 [kglrfcl()+492] [SIGSEGV] [ADDR:0x8] [PC:0x10815F0C] [Address not mapped to object] []
Errors in file /data1/oradata/diag/rdbms/oradb/oradb/trace/oradb_clmn_7090.trc  (incident=240029):
ORA-602 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /data1/oradata/diag/rdbms/oradb/oradb/incident/incdir_240029/oradb_clmn_7090_i240029.trc
2023-06-11T22:15:19.596187+08:00
Dumping diagnostic data in directory=[cdmp_20230611221519], requested by (instance=1, osid=7090 (CLMN)), summary=[incident=240028].
2023-06-11T22:15:19.693430+08:00
USER (ospid: 7090): terminating the instance due to error 602
2023-06-11T22:15:19.771330+08:00
opiodr aborting process unknown ospid (23847) as a result of ORA-1092
2023-06-11T22:15:20.945003+08:00
System state dump requested by (instance=1, osid=7090 (CLMN)), summary=[abnormal instance termination].
System State dumped to trace file /data1/oradata/diag/rdbms/oradb/oradb/trace/oradb_diag_7109_20230611221520.trc
2023-06-11T22:15:25.917215+08:00
Instance terminated by USER, pid = 7090
前台进程18036进程crash触发ora-07445错误，系统调用clmn（7090）进程去清理此进程的时候
clmn进程也出现了crash。最终实例宕机。           

查看oradb_ora_18036_i240700.trc发现call stack信息如下。

2023-06-11 22:15:12.775 :kjzduptcctx(): Notifying DIAG for crash event (incident=240700)
----- Abridged Call Stack Trace -----
ksedsts()+346<-kjzduptcctx()+868<-kjzdpcrshnfy()+380<-dbkedKstDump()+27<-dbgdaExecuteAction()+354<-dbgerRunAction()+108<-dbgerRunActions()+3719<-dbgexPhaseII()+1688<-dbgexExplicitEndInc()+602<-dbgeEndDDEInvocationImpl()+658<-ssexhd()+3274<-sslssSynchHdlr()+399
<-sslsshandler()+118<-__sighandler()<-kglrfcl()+492<-kglUnsetHandleReference()+120<-kglhdda()+777<-kghfreup()+172<-kghfrunp()+1505<-kghfnd()+984 
----- End of Abridged Call Stack Trace -----
----- END DDE Action: 'dumpKSTBuffers' (SUCCESS, 3 csec) -----
           

故障分析

2023-06-11T22:15:11.809567+08:00 出现异常告警ORA-07445  ，18036客户端进程异常
18036进程信息：
Unix process pid: 18036, image: oracle@TEST01
*** 2023-06-11T22:15:11.808756+08:00
*** SESSION ID:(1370.29059) 2023-06-11T22:15:11.808772+08:00
*** CLIENT ID:() 2023-06-11T22:15:11.808775+08:00
*** SERVICE NAME:(SYS$USERS) 2023-06-11T22:15:11.808778+08:00
*** MODULE NAME:(JDBC Thin Client) 2023-06-11T22:15:11.808781+08:00
*** ACTION NAME:() 2023-06-11T22:15:11.808784+08:00
*** CLIENT DRIVER:(jdbcthin) 2023-06-11T22:15:11.808786+08:00
clmn进程去清理失败进程的时候触发bug，导致自身也crash。
ORA-00602: 内部编程异常错误
ORA-07445: 出现异常错误: 核心转储 [kglrfcl()+492] [SIGSEGV] [ADDR:0x8] [PC:0x10815F0C] [Address not mapped to object] []
2023-06-11T22:15:19.291357+08:00
Incident 240029 created, dump file: /data1/oradata/diag/rdbms/oradb/oradb/incident/incdir_240029/oradb_clmn_7090_i240029.trc
ORA-602 [] [] [] [] [] [] [] [] [] [] [] []
2023-06-11 22:15:19.685 :kjzduptcctx(): Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+346<-kjzduptcctx()+868<-kjzdicrshnfy()+1113<-ksuitm_opt()+1678<-ksbrdp()+4494<-opirip()+609<-opidrv()+602<-sou2o()+145<-opimai_real()+202<-ssthrdmain()+417<-main()+262<-__libc_start_main()+245 
----- End of Abridged Call Stack Trace -----
*** 2023-06-11T22:15:19.693369+08:00
USER (ospid: 7090): terminating the instance due to error 602
ksuitm: waiting up to [5] seconds before killing DIAG(7109)
最终clmn（7090）进程终止实例
System State dumped to trace file /data1/oradata/diag/rdbms/oradb/oradb/trace/oradb_diag_7109_20230611221520.trc
2023-06-11T22:15:25.917215+08:00
Instance terminated by USER, pid = 7090
实例重启后恢复正常。           

后台进程 PMON, Cleanup Main Process (CLMN), 和 Cleanup Helper Processes (CLnn)。主要负责监控以及清理其他的进程，在某些情况下 PMON 进程组也负责对损坏的不可恢复的资源进行隔离，避免数据库实例关闭。  PMON 进程检测后台进程是否终止，并负责对必要的进程尝试恢复。PMON 会将清理工作委托给 CLMN 进程作为清理工作的主进程，CLMN 会周期性的去清理中断的进程、会话、事务、网络连接、空闲回话以及超时的网络连接和事务等。CLnn 协助 CLMN 清理中断的进程和会话。CLnn 可以存在多个，数量和当前需要清理的工作量和当前清理效率成正比。

查看mos后得出报错信息和此文档一致：Bug 29458132 Instance Might Crash With ORA-07445 [kglrfcl]

Affects:

Product (Component)

Oracle Server (Rdbms)

Range of versions believed to be affected

Versions BELOW 21.3

Versions confirmed as being affected

• 19.9.0
• 19.8.0
• 19.7.0
• 19.6.0
• 19.1.0
• 18.13.0
• 18.12.0
• 18.11.0
• 18.4.0
• 12.2.0.1 (Base Release)
• 12.1.0.2 (Server Patch Set)

Platforms affected

Generic (all / most platforms affected)

Fixed:

The fix for 29458132 is first included in

• 19.10.0.0.210119 (Jan 2021) Database Release Update (DB RU)
• 12.1.0.2.211019 (OCT 2021) Database Patch Set Update
• 12.1.0.2.211019 (OCT 2021) Database Proactive Bundle Patch

• 12.1.0.2.211019 (Oct 2021) Bundle Patch for Windows Platforms

  Interim patches may be available for earlier versions - click here to check.

Related To:

• Instance May Crash
• Process May Dump (ORA-7445) / Abend / Abort
• Memory Corruption
• Dump in or under kglrfcl
• Stack is likely to include kglrfcl
• (None Specified)

Description

A foreground process might crash with ORA-7445 [kglrfcl].  和故障的信息一致
CLMN tries to cleanup the dead process, but fails with the 
same error, hence instance crashes. 
 
Call stack might include: 
... kglrfcl kglUnsetHandleReference kglhdda ...  和故障的call stack信息一致
 
 REDISCOVERY INFORMATION:
 If a process fails with ORA-7445 [kglrfcl] and CLMN crashes 
 while cleaning the dead process, this bug might be hit