還原真實的 cache recovery

在maclean blog 上看到的這篇文章，很不錯，轉貼過來，原文連結如下：

http://www.oracledatabase12g.com/archives/whats-smon-enabling-cache-recovery.html

我們在學習Oracle基礎知識的時候會了解到執行個體恢複(Instance Recovery)或者說崩潰恢複(Crash recovery)的概念，有時候甚至于這2個名詞在我們日常的語言中表達同樣的意思。

實際上Instance Recovery與Crash Recovery是存在差別的：針對單執行個體(single instance)或者RAC中所有節點全部崩潰後的恢複，我們稱之為Crash Recovery。而對于RAC中的某一個節點失敗，存活節點(surviving instance)試圖對失敗節點線程上redo做應用的情況，我們稱之為Instance Recovery。

不管是Instance Recovery還是Crash Recovery，都由2個部分組成：cache recovery和transaction recovery。

根據官方文檔的介紹，Cache Recovery也叫Rolling Forward，就是我們常說的前滾；而Transaction Recovery也叫Rolling Back，就是我們常說的復原。

前滾：在redo log file中，從目前的start SCN開始，重做後面的已送出之事務。

復原：rollback未完成(dead transaction)事務。

我的Blog 也有關于這2個概念的說明：

Oracle 執行個體恢複時前滾（roll forward）後滾（roll back）問題

http://www.cndba.cn/Dave/article/1410

Crash Recovery測試:

SQL> shutdown abort;

ORACLE instance shut down.

SQL> startup mount;

ORACLE instance started.

Total System Global Area 1065353216 bytes

Fixed Size 2089336 bytes

Variable Size 486542984 bytes

Database Buffers 570425344 bytes

Redo Buffers 6295552 bytes

Database mounted.

SQL> alter database open;

Crash Recovery将從alter database open開始，我們來觀察其日志

==================alert.log====================

alter database open

Tue Jun 14 18:19:53 2011

Beginning crash recovery of 1 threads

parallel recovery started with 2 processes

Tue Jun 14 18:19:53 2011

Started redo scan

Tue Jun 14 18:19:53 2011

Completed redo scan

0 redo blocks read, 0 data blocks need recovery

Tue Jun 14 18:19:53 2011

Started redo application at

Thread 1: logseq 1004, block 1124, scn 17136185

Tue Jun 14 18:19:53 2011

Recovery of Online Redo Log: Thread 1 Group 2 Seq 1004 Reading mem 0

Mem# 0: /flashcard/oradata/G10R2/onlinelog/o1_mf_2_6v34jokt_.log

Mem# 1: /s01/flash_recovery_area/G10R2/onlinelog/o1_mf_2_6v34jotq_.log

Tue Jun 14 18:19:53 2011

Completed redo application

Tue Jun 14 18:19:53 2011

Completed crash recovery at

Thread 1: logseq 1004, block 1124, scn 17156186

0 data blocks read, 0 data blocks written, 0 redo blocks read

Tue Jun 14 18:19:53 2011

LGWR: STARTING ARCH PROCESSES

ARC0: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

ARC0 started with pid=16, OS id=7829

Tue Jun 14 18:19:53 2011

Thread 1 advanced to log sequence 1005 (thread open)

Thread 1 opened at log sequence 1005

Current log# 3 seq# 1005 mem# 0: /flashcard/oradata/G10R2/onlinelog/o1_mf_3_6v34jpmp_.log

Current log# 3 seq# 1005 mem# 1: /s01/flash_recovery_area/G10R2/onlinelog/o1_mf_3_6v34jpyn_.log

Successful open of redo thread 1

Tue Jun 14 18:19:53 2011

ARC0: Becoming the 'no FAL' ARCH

ARC0: Becoming the 'no SRL' ARCH

ARC0: Becoming the heartbeat ARCH

Tue Jun 14 18:19:53 2011

SMON: enabling cache recovery

Tue Jun 14 18:19:53 2011

db_recovery_file_dest_size of 204800 MB is 6.81% used. This is a

user-specified limit on the amount of space that will be used by this

database for recovery-related files, and does not reflect the amount of

space available in the underlying filesystem or ASM diskgroup.

Tue Jun 14 18:19:54 2011

Successfully onlined Undo Tablespace 1.

Tue Jun 14 18:19:54 2011

SMON: enabling tx recovery

Database Characterset is UTF8

Opening with internal Resource Manager plan

where NUMA PG = 1, CPUs = 2

replication_dependency_tracking turned off (no async multimaster replication found)

Starting background process QMNC

QMNC started with pid=17, OS id=7831

Tue Jun 14 18:19:55 2011

Completed: alter database open

注意上述單執行個體Crash Recovery到資料庫打開的整個過程:

（1）. alter database open

（2）. Beginning crash recovery of 1 threads

（3）. Started redo scan

（4）. Completed redo scan

（5）. Started redo application at

（6）. Completed redo application

（7）. Completed crash recovery at

（8）. SMON: enabling cache recovery

（9）. Successfully onlined Undo Tablespace 1

（10）. SMON: enabling tx recovery

（11）. Completed: alter database open

從上述步驟中我們可以看到三種恢複名詞，即：

（1）. crash recovery

（2）. cache recovery

（3）. tx recovery

這和官方文檔所描述的Crash Recovery概念是不一緻的，我們現在來理清這幾種recovery。

crash recovery包含對redo的scan和application，顯然其完成的是Rolling Forward前滾的工作，告警日志中出現的crash recovery等同于官方文檔中介紹的”cache recovery”，我們可以将” Completed crash recovery”看做前滾完成的标志。而tx recovery從字面就可以看出實際上是Transaction Recovery，tx recovery發生在Undo Tablespace online之後(復原事務的前提是Undo可用)，資料完成打開操作之前(“Completed: alter database open”)。注意tx recovery并不要求資料庫打開前完成，僅僅是在資料庫打開之前由smon啟動(“SMON: enabling tx recovery”)。

剩下的唯一的問題是，這裡的cache recovery是什麼？顯然它不是官方文檔中所描述的”cache recovery”，幾乎沒有任何文檔介紹存在這樣一個recovery操作，這也是本文重點要介紹的。

我們來看另一個示範，這個示範用以說明cache recovery還存在于最普通的不包含Crash Recovery的資料庫打開過程中:

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup mount;

ORACLE instance started.

Total System Global Area 1065353216 bytes

Fixed Size 2089336 bytes

Variable Size 486542984 bytes

Database Buffers 570425344 bytes

Redo Buffers 6295552 bytes

Database mounted.

SQL> alter database open;

Database altered.

SQL> select * from v$version;

BANNER

----------------------------------------------------------------

Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bi

PL/SQL Release 10.2.0.4.0 - Production

CORE 10.2.0.4.0 Production

TNS for Linux: Version 10.2.0.4.0 - Production

NLSRTL Version 10.2.0.4.0 - Production

SQL> select * from global_name;

GLOBAL_NAME

--------------------------------------------------------------------------------

www.oracledatabase12g.com

==================alert.log====================

alter database open

Tue Jun 14 18:43:52 2011

LGWR: STARTING ARCH PROCESSES

ARC0: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

ARC0 started with pid=14, OS id=8133

Tue Jun 14 18:43:52 2011

Thread 1 opened at log sequence 1005

Current log# 3 seq# 1005 mem# 0: /flashcard/oradata/G10R2/onlinelog/o1_mf_3_6v34jpmp_.log

Current log# 3 seq# 1005 mem# 1: /s01/flash_recovery_area/G10R2/onlinelog/o1_mf_3_6v34jpyn_.log

Successful open of redo thread 1

Tue Jun 14 18:43:52 2011

ARC0: Becoming the 'no FAL' ARCH

ARC0: Becoming the 'no SRL' ARCH

ARC0: Becoming the heartbeat ARCH

Tue Jun 14 18:43:52 2011

SMON: enabling cache recovery

Tue Jun 14 18:43:53 2011

Successfully onlined Undo Tablespace 1.

Tue Jun 14 18:43:53 2011

SMON: enabling tx recovery

Database Characterset is UTF8

Opening with internal Resource Manager plan

where NUMA PG = 1, CPUs = 2

replication_dependency_tracking turned off (no async multimaster replication found)

Tue Jun 14 18:43:53 2011

Incremental checkpoint up to RBA [0x3ed.624.0], current log tail at RBA [0x3ed.944.0]

Tue Jun 14 18:43:53 2011

Starting background process QMNC

QMNC started with pid=15, OS id=8135

Tue Jun 14 18:43:53 2011

Completed: alter database open

因為是clean shutdown，是以這裡不存在crash recovery。但這裡同樣出現了”SMON: enabling cache recovery”，可見cache recovery是每次執行個體啟動instance startup必要執行的一種恢複操作。但問題是，這個恢複操作到底針對何種對象？

實際上cache recovery所要恢複的是rowcache，也就是我們常說的字典緩存(dictionary cache)。關于這個結論，肯定有很多人要問我這樣說的依據是什麼，對應于這個”cache recovery”的問題，我們很難從google中得到一些啟示，因為它和官方文檔所描述的”cache recovery-rolling forward”存在重名的關系。

為了證明cache recovery所恢複的是row cache，我們需要一個實證，從正式的系統中得到驗證。要做到這一點是比較困難的，我們需要Oracle願意把整個database open的過程變成慢動作來供我們參考，驗證要用到一些調試工具，例如gdb或者dbx。

我們首先将執行個體啟動到mount狀态，并對執行startup的LOCAL程序做gdb的breakpoint斷點調試:

SQL> shutdown abort;

ORACLE instance shut down.

SQL> startup mount;

ORACLE instance started.

Total System Global Area 1065353216 bytes

Fixed Size 2089336 bytes

Variable Size 486542984 bytes

Database Buffers 570425344 bytes

Redo Buffers 6295552 bytes

Database mounted.

找出LOCAL程序的系統程序号SPID

SQL> select spid from v$process

2 where addr in (

3 select paddr from v$session

4 where sid=(select distinct sid from v$mystat))

5 /

SPID

------------

8326

在執行個體startup nomount/mount後共享池的library cache就是可用的

SQL> select namespace from v$librarycache where gets!=0;

NAMESPACE

---------------

SQL AREA

TABLE/PROCEDURE

而rowcache則尚未被填充，因為字典緩存來源于自對象(bootstrap$)和字典基表

SQL> select parameter,count,gets from v$rowcache where count!=0;

no rows selected

另開一個terminal視窗，并執行對LOCAL程序8326的gdb breakpoint調試

[[email protected] ~]$ gdb $ORACLE_HOME/bin/oracle 8326

GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5)

License GPLv3+: GNU GPL version 3 or later

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law. Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

...

Reading symbols from /s01/db_1/bin/oracle...(no debugging symbols found)...done.

Attaching to program: /s01/db_1/bin/oracle, process 8326

Reading symbols from /s01/db_1/lib/libskgxp10.so...(no debugging symbols found)...done.

Loaded symbols for /s01/db_1/lib/libskgxp10.so

Reading symbols from /s01/db_1/lib/libhasgen10.so...(no debugging symbols found)...done.

Loaded symbols for /s01/db_1/lib/libhasgen10.so

Reading symbols from /s01/db_1/lib/libskgxn2.so...(no debugging symbols found)...done.

Loaded symbols for /s01/db_1/lib/libskgxn2.so

Reading symbols from /s01/db_1/lib/libocr10.so...(no debugging symbols found)...done.

Loaded symbols for /s01/db_1/lib/libocr10.so

Reading symbols from /s01/db_1/lib/libocrb10.so...(no debugging symbols found)...done.

Loaded symbols for /s01/db_1/lib/libocrb10.so

Reading symbols from /s01/db_1/lib/libocrutl10.so...(no debugging symbols found)...done.

Loaded symbols for /s01/db_1/lib/libocrutl10.so

Reading symbols from /s01/db_1/lib/libjox10.so...(no debugging symbols found)...done.

Loaded symbols for /s01/db_1/lib/libjox10.so

Reading symbols from /s01/db_1/lib/libclsra10.so...(no debugging symbols found)...done.

Loaded symbols for /s01/db_1/lib/libclsra10.so

Reading symbols from /s01/db_1/lib/libdbcfg10.so...(no debugging symbols found)...done.

Loaded symbols for /s01/db_1/lib/libdbcfg10.so

Reading symbols from /s01/db_1/lib/libnnz10.so...(no debugging symbols found)...done.

Loaded symbols for /s01/db_1/lib/libnnz10.so

Reading symbols from /usr/lib64/libaio.so.1...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libaio.so.1

Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libdl.so.2

Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libm.so.6

Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.

[Thread debugging using libthread_db enabled]

Loaded symbols for /lib64/libpthread.so.0

Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnsl.so.1

Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libc.so.6

Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/ld-linux-x86-64.so.2

Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnss_files.so.2

0x0000003181a0d8e0 in __read_nocancel () from /lib64/libpthread.so.0

輸入斷點kcrf_commit_force和kqlobjlod

(gdb) break kcrf_commit_force

Breakpoint 1 at 0x2724b6c

(gdb) break kqlobjlod

Breakpoint 2 at 0x1ac5e8c

在之前的terminal中執行資料庫打開操作，因為breakpoint的關系這個open操作會hang住，這時我們通過觀察告警日志來了解恢複進度

SQL> alter database open; --這裡會hang住

在gdb下輸入continue，

(gdb) c

Continuing.

Breakpoint 1, 0x0000000002724b6c in kcrf_commit_force ()

觀察告警日志可以發現redo application已經完成，但還未進入cache recovery階段

alter database open

Tue Jun 14 19:14:33 2011

Beginning crash recovery of 1 threads

parallel recovery started with 2 processes

Tue Jun 14 19:14:33 2011

Started redo scan

Tue Jun 14 19:14:33 2011

Completed redo scan

39 redo blocks read, 74 data blocks need recovery

Tue Jun 14 19:14:33 2011

Started redo application at

Thread 1: logseq 1006, block 1155

Tue Jun 14 19:14:33 2011

Recovery of Online Redo Log: Thread 1 Group 1 Seq 1006 Reading mem 0

Mem# 0: /flashcard/oradata/G10R2/onlinelog/o1_mf_1_6v34jnkn_.log

Mem# 1: /s01/flash_recovery_area/G10R2/onlinelog/o1_mf_1_6v34jnst_.log

Tue Jun 14 19:14:33 2011

Completed redo application

Tue Jun 14 19:14:33 2011

Completed crash recovery at

Thread 1: logseq 1006, block 1194, scn 17200193

74 data blocks read, 74 data blocks written, 39 redo blocks read

Tue Jun 14 19:14:33 2011

LGWR: STARTING ARCH PROCESSES

ARC0: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

ARC0 started with pid=17, OS id=8656

Tue Jun 14 19:14:33 2011

Thread 1 advanced to log sequence 1007 (thread open)

Thread 1 opened at log sequence 1007

Current log# 2 seq# 1007 mem# 0: /flashcard/oradata/G10R2/onlinelog/o1_mf_2_6v34jokt_.log

Current log# 2 seq# 1007 mem# 1: /s01/flash_recovery_area/G10R2/onlinelog/o1_mf_2_6v34jotq_.log

Successful open of redo thread 1

Tue Jun 14 19:14:33 2011

ARC0: Becoming the 'no FAL' ARCH

ARC0: Becoming the 'no SRL' ARCH

ARC0: Becoming the heartbeat ARCH

db_recovery_file_dest_size of 204800 MB is 6.81% used. This is a

user-specified limit on the amount of space that will be used by this

database for recovery-related files, and does not reflect the amount of

space available in the underlying filesystem or ASM diskgroup.

Tue Jun 14 19:14:37 2011

Incremental checkpoint up to RBA [0x3ef.3.0], current log tail at RBA [0x3ef.3.0]

且此時rowcache仍未被填充

SQL> select parameter,count,gets from v$rowcache where count!=0;

no rows selected

在gdb界面下再次執行continue 2次

(gdb) c

Continuing.

Breakpoint 1, 0x0000000002724b6c in kcrf_commit_force ()

(gdb) c

Continuing.

Breakpoint 2, 0x0000000001ac5e8c in kqlobjlod ()

觀察告警日志可以發現已開始cache recovery，但也卡陷在cache recovery上，這保證我們的示範不受騷擾

Tue Jun 14 19:16:44 2011

SMON: enabling cache recovery

此時rowcache中出現唯一的一個dc_objects對象

select parameter,count,gets from v$rowcache where count!=0;

PARAMETER COUNT GETS

-------------------------------- ---------- ----------

dc_objects 1 1

這個對象是什麼呢？也許你已經猜到了，我們做一個rowcache dump來看一下:

SQL> ALTER SESSION SET EVENTS 'immediate trace name row_cache level 10';

================row_cache trace===================

BUCKET 43170:

row cache parent object: address=0x92326060 cid=8(dc_objects)

hash=f3d1a8a1 typ=11 transaction=(nil) flags=00000001

own=0x92326130[0x9230f628,0x9230f628] wat=0x92326140[0x92326140,0x92326140] mode=S

status=EMPTY/-/-/-/-/-/-/-/-

set=0, complete=FALSE

data=

00000000 4f42000a 5453544f 24504152 00000000 00000000 00000000 00000000

00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000

00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

00000000 00000000

BUCKET 43170 total object count=1

可以看到上述dc_objects尚未完成加載(status=EMPTY & complete=FALSE )，那麼這是一個什麼object呢？

4f42000a 5453544f 24504152 => 轉換為文本即：OB TSTO$PAR也就是BOOTSTRAP$

換而言之在cache recovery時第一個恢複的字典緩存對象是BOOTSTRAP$，這并不出乎我們的意料。

啟動執行個體的LOCAL程序的等待事件為instance state change，這是正常情況下我們觀察不到得

SQL> select event,p1text,p1 from v$session where wait_class!='Idle';

EVENT                                    P1TEXT                                           P1

---------------------------------------- ---------------------------------------- ----------

instance state change                    layer                                             2

在gdb界面下再次continue，将載入更多的rowcache對象

(gdb) c

Continuing.

Breakpoint 2, 0x0000000001ac5e8c in kqlobjlod ()

BUCKET 37:

row cache parent object: address=0x916cd980 cid=3(dc_rollback_segments)

hash=5fed2a24 typ=9 transaction=(nil) flags=000000a6

own=0x916cda50[0x916cda50,0x916cda50] wat=0x916cda60[0x916cda60,0x916cda60] mode=N

status=VALID/INSERT/-/FIXED/-/-/-/-/-

data=

00000000 00000000 00000001 00000009 59530006 4d455453 00000000 00000000

00000000 00000000 00000000 00000000 00000003 00000000 00000000 00000000

00000000 00000000 00000000 00000000

BUCKET 37 total object count=1

595300064d455453 -> SYSTEM 屬于dc_rollback_segments 也就是著名的system復原段

BUCKET 55556:

row cache parent object: address=0x916d8cd0 cid=8(dc_objects)

hash=ce89d903 typ=11 transaction=(nil) flags=00000001

own=0x916d8da0[0x9230f628,0x9230f628] wat=0x916d8db0[0x916d8db0,0x916d8db0] mode=S

status=EMPTY/-/-/-/-/-/-/-/-

set=0, complete=FALSE

data=

00000000 5f430006 234a424f 00000000 00000000 00000000 00000000 00000000

00000000 00000005 00000000 00000000 00000000 00000000 00000000 00000000

00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

00000000 00000000

BUCKET 55556 total object count=1

5f430006 234a424f -> C_OBJ# 是著名的bootstrap對象之一，可以從$ORACLE_HOME/rdbms/admin/sql.bsq中找到它

create rollback segment SYSTEM tablespace SYSTEM

storage (initial 50K next 50K)

create cluster c_obj# (obj# number)

pctfree 5 size 800

storage (initial 130K next 200k maxextents unlimited pctincrease 0)

我們還可以通過v$rowcache_parent視圖來了解dictionary cache的情況

SQL> col cache_name for a20

SQL> col keystr for a31

SQL> set linesize 200

SQL> select address,cache_name,existent,lock_mode,saddr,substr(key,1,30) keystr from v$rowcache_parent;

ADDRESS          CACHE_NAME           E  LOCK_MODE SADDR            KEYSTR

---------------- -------------------- - ---------- ---------------- -------------------------------

00000000916CCE20 dc_tablespaces       N          0 00               000000000000000000000000000000

00000000916CD980 dc_rollback_segments Y          0 00               000000000000000000000000000000

0000000092326060 dc_objects           Y          0 00               000000000A00424F4F545354524150

00000000916D8CD0 dc_objects           N          3 000000009BD91328 000000000600435F4F424A23000000

00000000916DA830 dc_object_ids        Y          0 00               380000000000000000000000000000

可以看到持有row cache lock的會話是'000000009BD91328'，且該dc_objects對象還處于non-existent狀态，換而言之真正裝載rowcache的是啟動執行個體的LOCAL服務程序

SQL> select sid,program,event,p1,p2,p3 from v$session where saddr='000000009BD91328';

SID PROGRAM                                          EVENT                                    P1   P2 P3

----- ------------------------------------------------ ---------------------------------------- -- ---- --

3294 [email protected] (TNS V1-V3)               db file scattered read                    1  378  3

該程序正在等待db file scattered read，fileid->1,block-378，這些塊屬于BOOTSTRAP$表

BOOTSTRAP$對象已從rowcache被載入到library cache中

SQL> select kglhdadr,kglnaobj from x$kglob where kglobtyp=2 and kglnaobj not like 'X$%';

KGLHDADR             KGLNAOBJ

-------------------- --------------------

0000000092326990     BOOTSTRAP$

SQL> select owner||'.'||Name from v$db_object_cache where type='TABLE' and name not like 'X$%';

OWNER||'.'||NAME

--------------------------------------------------------------------------------

SYS.BOOTSTRAP$

初步總結：

（1）在資料庫正式open前需要恢複字典緩存，這個步驟被稱為cache recovery，其實是row cache recovery。與官方文檔中描述的”cache recovery”不同，row cache recovery應當是Oracle Internal的叫法。

（2）實際執行row cache recovery的不是SMON程序，而是啟動執行個體的服務程序

-------------------------------------------------------------------------------------------------------

QQ:492913789

Email:[email protected]

Blog: http://www.cndba.cn/dave

DBA1 群：62697716(滿); DBA2 群：62697977(滿) DBA3 群：62697850(滿)

DBA 超級群：63306533(滿); DBA4 群： 83829929 DBA5群： 142216823

DBA6 群：158654907 聊天群：40132017 聊天2群：69087192

--加群需要在備注說明Oracle表空間和資料檔案的關系，否則拒絕申請

還原真實的 cache recovery

繼續閱讀

timesten系列五：如何定義cache，和背景oracle資料庫同步資料

Intel® 64 and IA-32 Architectures Software Developer's Manuals

Partitioning Concepts

go協程池（goroutine）、go線程池（Thread）、go連接配接池

MySQL安裝時常見問題和解決方案【詳解】

西部資料的新品PCIe4.0SSD固态硬碟WDBlueSN580，速度真的不錯！1TB順序讀取速度高達4150MB/s6

HP Proliant 系列伺服器使用 SmartStart CD光牒配置陣列卡過程

Doris資料更新及删除1.資料更新

[crash分析][arm]應用程式Segmentation fault

使用JDeveloper Remote debug PLSQL程式

Debug時檢測到Loaderlock的解決辦法

Communications link failure Last packet sent to the server was 0 ms ago

單字母域名

使用Windbg調試.Net應用程式

C#多線程——前台線程和背景線程

testlink 切換資料庫