ARC0 Encountered disk IO error 19502

一個二級客戶的一個備份庫,是以平時沒怎麼關注，今天去處理ntpdbug的時候想到順便看看oracle運作情況，結果一看告警日志一大堆錯誤資訊，顯示歸檔程序無法完成：

Fri Jun 22 22:00:22 CST 2012

ARC0: Encountered disk I/O error 19502

Fri Jun 22 22:00:22 CST 2012

ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: '/u01/oracle/10g/arch/1_507_780753325.dbf' (error 19502)

 (<ORACLE-SID>)

Fri Jun 22 22:00:22 CST 2012

Errors in file /u01/oracle/10g/admin/<ORACLE-SID>/bdump/<ORACLE-SID>_arc0_28109.trc:

ORA-19502: Message 19502 not found; No message file for product=RDBMS, facility=ORA; arguments: [/u01/oracle/10g/arch/1_507_780753325.dbf] [376833] [512]

ORA-27072: Message 27072 not found; No message file for product=RDBMS, facility=ORA

Linux-x86_64 Error: 9: Bad file descriptor

Additional information: 4

Additional information: 376833

Additional information: 581120

ORA-19502: Message 19502 not found; No message file for product=RDBMS, facility=ORA; arguments: [/u01/oracle/10g/arch/1_507_780753325.dbf] [376833] [512]

Fri Jun 22 22:00:22 CST 2012

Errors in file /u01/oracle/10g/admin/<ORACLE-SID>/bdump/<ORACLE-SID>_arc0_28109.trc:

ORA-19502: Message 19502 not found; No message file for product=RDBMS, facility=ORA; arguments: [/u01/oracle/10g/arch/1_507_780753325.dbf] [376833] [512]

ORA-27072: Message 27072 not found; No message file for product=RDBMS, facility=ORA

Linux-x86_64 Error: 9: Bad file descriptor

Additional information: 4

Additional information: 376833

Additional information: 581120

ORA-19502: Message 19502 not found; No message file for product=RDBMS, facility=ORA; arguments: [/u01/oracle/10g/arch/1_507_780753325.dbf] [376833] [512]

ARC0: I/O error 19502 archiving log 3 to '/u01/oracle/10g/arch/1_507_780753325.dbf'

ARCH: Archival stopped, error occurred. Will continue retrying

Fri Jun 22 22:00:22 CST 2012

ORACLE Instance <ORACLE-SID> - Archival Error

Fri Jun 22 22:00:22 CST 2012

ORA-16038: Message 16038 not found; No message file for product=RDBMS, facility=ORA; arguments: [3] [507]

ORA-19502: Message 19502 not found; No message file for product=RDBMS, facility=ORA

ORA-00312: Message 312 not found; No message file for product=RDBMS, facility=ORA; arguments: [3] [1] [/u01/oracle/10g/oradata/<ORACLE-SID>/redo03.log]

Fri Jun 22 22:00:22 CST 2012

Errors in file /u01/oracle/10g/admin/<ORACLE-SID>/bdump/<ORACLE-SID>_arc0_28109.trc:

ORA-16038: Message 16038 not found; No message file for product=RDBMS, facility=ORA; arguments: [3] [507]

ORA-19502: Message 19502 not found; No message file for product=RDBMS, facility=ORA

ORA-00312: Message 312 not found; No message file for product=RDBMS, facility=ORA; arguments: [3] [1] [/u01/oracle/10g/oradata/<ORACLE-SID>/redo03.log]

Fri Jun 22 22:00:26 CST 2012

ARCH: Archival stopped, error occurred. Will continue retrying

Fri Jun 22 22:00:26 CST 2012

ORACLE Instance <ORACLE-SID> - Archival Error

Fri Jun 22 22:00:26 CST 2012

ORA-16014: Message 16014 not found; No message file for product=RDBMS, facility=ORA; arguments: [3] [507]

ORA-00312: Message 312 not found; No message file for product=RDBMS, facility=ORA; arguments: [3] [1] [/u01/oracle/10g/oradata/<ORACLE-SID>/redo03.log]

Fri Jun 22 22:00:26 CST 2012

Errors in file /u01/oracle/10g/admin/<ORACLE-SID>/bdump/<ORACLE-SID>_arc1_28111.trc:

ORA-16014: Message 16014 not found; No message file for product=RDBMS, facility=ORA; arguments: [3] [507]

ORA-00312: Message 312 not found; No message file for product=RDBMS, facility=ORA; arguments: [3] [1] [/u01/oracle/10g/oradata/<ORACLE-SID>/redo03.log]

重要資訊在：Linux-x86_64 Error: 9: Bad file descriptor，這個是作業系統的異常錯誤，馬上想到了磁盤空間，果然空間不夠了：

[[email protected] bdump]$ df -h

檔案系統              容量  已用 可用 已用% 挂載點

/dev/cciss/c0d0p2     128G  121G     0 100% /

/dev/cciss/c0d0p1     193M   22M  162M  12% /boot

tmpfs                 7.9G     0  7.9G   0% /dev/shm

這個備庫本來就沒必要開啟歸檔，将其關閉即可：

SQL> shutdown immediate;

ORA-01109: database not open

Database dismounted.

ORACLE instance shut down.

SQL> startup mount

ORACLE instance started.

Total System Global Area 1610612736 bytes

Fixed Size            2096632 bytes

Variable Size          805306888 bytes

Database Buffers      788529152 bytes

Redo Buffers           14680064 bytes

Database mounted.

SQL> alter database noarchivelog;

Database altered.

SQL> alter database open;

Database altered.

SQL> archive log list;

Database log mode           No Archive Mode

Automatic archival           Disabled

Archive destination           /u01/oracle/10g/arch

Oldest online log sequence     514

Current log sequence           516

并将其歸檔檔案删除。

-The End-