Exadata關于磁盤部分，新增加了lun，celldisk，griddisk的概念，這裡簡單介紹一下，如下圖：
<a target="_blank" href="http://blog.51cto.com/attachment/201302/164320203.jpg"></a>
我們可以看到一個實體硬碟添加到Cell之後，會自動建立LUN，LUN對應的是建立一個celldisk，一個celldisk對應一個或者多個griddisk，而griddisk就可以提供給ASM執行個體使用了。
這裡我主要記錄一下我在将一個正在運作的Exadata機器， 它的一個磁盤正常删除并添加的完整過程。
(這裡強調一下“CellCLI>”都是在Cell上執行的指令，“SQL>” 都是在compute node（既資料庫節點）上執行的指令)
1.首先要确定我需要删除的硬碟的LUN的位置，celldisk名字，通過下面指令可以查詢
CellCLI> LIST CELLDISK WHERE name LIKE 'CD_03.*' detail
name: CD_03_dmorlcel08 <<<<<<<<< celldisk名字
comment:
creationTime: 2012-04-20T04:37:54-04:00
deviceName: /dev/sdd
devicePartition: /dev/sdd
diskType: HardDisk
errorCount: 0
freeSpace: 0
id: 772cc8b1-ca81-468e-8b65-16e2724ef4da
interleaving: none
lun: 0_3 <<<<<<<<< 在0_3 這個LUN
raidLevel: 0
size: 557.859375G
status: normal
2.查詢griddisk的資訊，這個可以找到asm中的disk name，以及diskgroup的name。
通過這個指令還可以看到這個celldisk對應三個griddisk。
CellCLI> LIST GRIDDISK WHERE cellDisk LIKE 'CD_03.*' detail
name: DATA_CD_03_dmorlcel08
asmDiskGroupName: DATA
asmDiskName: DATA_CD_03_DMORLCEL08 <<<<<<<<<<<<<<<<<<<<<<<<<ASM disk name
availableTo:
cellDisk: CD_03_dmorlcel08
creationTime: 2012-04-20T04:39:48-04:00
id: 524f59b3-32af-4926-bc40-bf094d05a71d
offset: 32M
size: 423G
status: active
name: DBFS_DG_CD_03_dmorlcel08
asmDiskGroupName: DBFS_DG
asmDiskName: DBFS_DG_CD_03_DMORLCEL08 <<<<<<<<<<<<<<<<<<<<<<<<<ASM disk name
creationTime: 2012-04-20T04:57:22-04:00
id: 001ef1df-5d41-4fab-b1ad-5a8983a63b8c
offset: 528.734375G
size: 29.125G
name: RECO_CD_03_dmorlcel08
asmDiskGroupName: RECO <<<<<<<<<<<<<<<<<<<<<<<<<ASM diskgroup name
asmDiskName: RECO_CD_03_DMORLCEL08 <<<<<<<<<<<<<<<<<<<<<<<<<ASM disk name
creationTime: 2012-04-20T04:56:28-04:00
id: 544c54c0-2969-4ad3-bea0-f3e2b4f1bf31
offset: 423.046875G
size: 105.6875G
3.通過ASM執行個體查詢磁盤資訊
$more /etc/oratab
#Backup file is /u01/app/oracle/product/11.2.0.3/db_home1/srvm/admin/oratab.bak.dmorldb05 line added by Agent
+ASM1:/u01/app/11.2.0.3/grid:N # line added by Agent
$ export ORACLE_SID=+ASM1
$ export ORACLE_HOME=/u01/app/11.2.0.3/grid
$ sqlplus / as sysasm
SQL> set line 400
SQL> select GROUP_NUMBER,DISK_NUMBER,LABEL,NAME,MOUNT_STATUS,state,FAILGROUP,MODE_STATUS,PATH from v$asm_disk where name='RECO_CD_03_DMORLCEL08'; <<<<<<<disk name need to correct
GROUP_NUMBER DISK_NUMBER LABEL NAME PATH
------------ ----------- ------------------------------- ------------------------------
4 10 RECO_CD_03_DMORLCEL08 RECO_CD_03_DMORLCEL08 o/192.168.10.16/RECO_CD_03_dmorlcel08
查詢對應的磁盤組
SQL> select GROUP_NUMBER,NAME from v$asm_diskgroup; <<<<<determine the diskgroup number
GROUP_NUMBER NAME
------------ ------------------------------
1 DATA
2 DBFS_DG
4 RECO
SQL> select NAME,ALLOCATION_UNIT_SIZE,STATE,TOTAL_MB,OFFLINE_DISKS,TOTAL_MB,FREE_MB,REQUIRED_MIRROR_FREE_MB from v$asm_diskgroup;
4.下面開始我們的删除操作，如果删除celldisk，首先應該從最上層，ASM層面開始删除磁盤
SQL> alter diskgroup RECO drop disk RECO_CD_03_DMORLCEL08 REBALANCE power 8;
SQL> alter diskgroup DBFS_DG drop disk DBFS_DG_CD_03_DMORLCEL08 REBALANCE power 8;
SQL> alter diskgroup DATA drop disk DATA_CD_03_DMORLCEL08 REBALANCE power 8;
5.删除的過程中，通過這個指令查收reblance進度，確定ASM删除磁盤的動作完成，再進行下一步操作。
SQL> set line 200
SQL> select * from v$asm_operation;
GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ----------
1 REBAL RUN 8 8 58593 207611 7586 19
SQL> /
GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE
1 REBAL RUN 8 8 58856 207656 7643 19
no rows selected <<<<<<<<<<<<表示reblance已經完成
6.ASM中删除掉DBFS_DG_CD_03_DMORLCEL08磁盤後，通過griddisk指令查詢，發現對應的asmDiskName，asmDiskGroupName，已經沒有值了。
CellCLI> list GRIDDISK DBFS_DG_CD_03_DMORLCEL08 detail
asmDiskGroupName:
asmDiskName:
status: active
7.删除griddisk，并查詢，發現已經查詢不到griddisk DBFS_DG_CD_03_DMORLCEL08的資訊
CellCLI> DROP GRIDDISK DBFS_DG_CD_03_DMORLCEL08;
GridDisk DBFS_DG_CD_03_dmorlcel08 successfully dropped
CellCLI> list GRIDDISK DBFS_DG_CD_03_DMORLCEL08 detail
CELL-02007: Grid disk does not exist: DBFS_DG_CD_03_DMORLCEL08
CellCLI> LIST gridDISK WHERE cellDisk LIKE '.*CD_03.*' detail
asmDiskName: DATA_CD_03_DMORLCEL08
8.繼續删除RECO_CD_03_dmorlcel08并查詢
CellCLI> DROP GRIDDISK RECO_CD_03_dmorlcel08;
GridDisk RECO_CD_03_dmorlcel08 successfully dropped
CellCLI> LIST gridDISK WHERE cellDisk LIKE '.*CD_03.*' detail
9.繼續删除DATA_CD_03_DMORLCEL08，并查詢
CellCLI> DROP GRIDDISK DATA_CD_03_DMORLCEL08
GridDisk DATA_CD_03_dmorlcel08 successfully dropped
CellCLI> LIST gridDISK WHERE cellDisk LIKE '.*CD_03.*' detail
name: CD_03_dmorlcel08
freeSpace: 557.8125G
freeSpaceMap: offset=32M,size=19.96875G
offset=20.015625G,size=537.84375G
lun: 0_3
CellCLI> drop celldisk CD_03_dmorlcel08
CellDisk CD_03_dmorlcel08 successfully dropped
10.删除celldisk，并查詢LUN資訊
CellCLI> LIST CELLDISK WHERE name LIKE 'CD_03.*' detail
CellCLI> LIST LUN 0_3 DETAIL
name: 0_3
cellDisk:
id: 0_3
isSystemLun: FALSE
lunAutoCreate: FALSE
lunSize: 557.861328125G
lunUID: 0_3
physicalDrives: 28:3
lunWriteCacheMode: "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU"
status: normal
比較一個正常的LUN資訊
CellCLI> LIST LUN 0_4 DETAIL
name: 0_4
cellDisk: CD_04_dmorlcel08 <<<<<<<
deviceName: /dev/sde
id: 0_4
lunUID: 0_4
physicalDrives: 28:4
11.在删除完成後，我們要開始添加的過程，實際troubleshooting中，會遇到Exadata在磁盤更換完成後，并沒有正常的建立celldisk和griddisk，我們需要手動建立celldisk和griddisk。
手動建立之前，我們需要查詢physicaldisk狀态，确認physicalSerial和你更換的磁盤是對應的
CellCLI> list physicaldisk where luns=0_3 detail
name: 28:3
deviceId: 24
enclosureDeviceId: 28
errMediaCount: 0
errOtherCount: 0
foreignState: false
luns: 0_3
makeModel: "SEAGATE ST360057SSUN600G"
physicalFirmware: 0A25
physicalInsertTime: 2012-01-20T15:57:20-05:00
physicalInterface: sas
physicalSerial: E12EQY
physicalSize: 558.9109999993816G
slotNumber: 3
12.建立celldisk，根據前面的資訊，确認需要建立的celldisk與LUN能準确的比對上，不要寫錯
CellCLI> create celldisk CD_03_dmorlcel08 lun=0_4（如果沒有正确比對，會遇到如下錯誤）
CELL-04527: Cannot complete the creation of cell disk CD_03_dmorlcel08. Received error: CELL-04522: The LUN 0_4 has a valid celldisk.
Cell disks are not created: CD_03_dmorlcel08
成功建立
CellCLI> create celldisk CD_03_dmorlcel08 lun=0_3
CellDisk CD_03_dmorlcel08 successfully created
14.首先查詢一個正常的磁盤，确認後續需要設定的資訊，如name,size,offset
CellCLI> list griddisk where celldisk=CD_04_dmorlcel08 attributes name,size,offset
DATA_CD_04_dmorlcel08 423G 32M
DBFS_DG_CD_04_dmorlcel08 29.125G 528.734375G
RECO_CD_04_dmorlcel08 105.6875G 423.046875G
15.建立griddisk磁盤
CellCLI> create griddisk DATA_CD_03_dmorlcel08 celldisk=CD_03_dmorlcel08,size=423G
CellCLI> create griddisk DBFS_DG_CD_03_dmorlcel08 celldisk=CD_03_dmorlcel08,size=29.125G
CellCLI> create griddisk RECO_CD_03_dmorlcel08 celldisk=CD_03_dmorlcel08,size=105.6875G
下面是執行的記錄，以及查詢結果
GridDisk DATA_CD_03_dmorlcel08 successfully created
GridDisk DBFS_DG_CD_03_dmorlcel08 successfully created
CellCLI> create griddisk RECO_CD_03_dmorlcel08 celldisk=CD_03_dmorlcel08,size=105.6875G
GridDisk RECO_CD_03_dmorlcel08 successfully created
可以查詢到celldisk資訊
CellCLI> LIST CELLDISK WHERE name LIKE 'CD_03.*' detail
creationTime: 2013-02-07T02:13:17-05:00
id: 2f3e651e-6fb6-47e7-a227-692645519ce4
可以查詢到griddisk資訊
creationTime: 2013-02-07T02:14:17-05:00
id: 6f9cb08e-fd8f-4063-b902-439387237f5d
creationTime: 2013-02-07T02:14:26-05:00
id: 055710ce-216a-4b10-a215-b38763ea65df
creationTime: 2013-02-07T02:14:33-05:00
id: fdf9a255-0a0d-4152-a557-de80b112ddff
offset: 452.171875G
CellCLI> list griddisk where celldisk=CD_03_dmorlcel08 attributes name,size,offset
DATA_CD_03_dmorlcel08 423G 32M
DBFS_DG_CD_03_dmorlcel08 29.125G 423.046875G
RECO_CD_03_dmorlcel08 105.6875G 452.171875G
16.如果磁盤沒有删除的情況下，需要force再次删除一下，但是我們的測試中，前面已經正常删除磁盤 .
SQL> alter diskgroup data drop disk DATA_CD_07_dm03cel13 force;
select GROUP_NUMBER,DISK_NUMBER,LABEL,NAME,MOUNT_STATUS,state,FAILGROUP,MODE_STATUS,PATH from v$asm_disk where name like '%CD_03_DMORLCEL08';
17.添加磁盤
SQL> ALTER DISKGROUP DATA ADD DISK 'o/192.168.10.16/DATA_CD_03_dmorlcel08' rebalance power 10;
SQL> ALTER DISKGROUP DBFS_DG ADD DISK 'o/192.168.10.16/DBFS_DG_CD_03_dmorlcel08' rebalance power 10;
SQL> ALTER DISKGROUP RECO ADD DISK 'o/192.168.10.16/RECO_CD_03_dmorlcel08' rebalance power 10;
添加完成兩個磁盤後，查詢ASM的結果
GROUP_NUMBER DISK_NUMBER LABEL NAME MOUNT_S STATE FAILGROUP MODE_ST PATH
------------ ----------- ------------------------------- ------------------------------ ------- -------- ------------------------------ ------- ------------------
1 3 DATA_CD_03_DMORLCEL08 DATA_CD_03_DMORLCEL08 CACHED NORMAL DMORLCEL08 ONLINE o/192.168.10.16/DATA_CD_03_dmorlcel08
2 6 DBFS_DG_CD_03_DMORLCEL08 DBFS_DG_CD_03_DMORLCEL08 CACHED NORMAL DMORLCEL08 ONLINE o/192.168.10.16/DBFS_DG_CD_03_dmorlcel08
18.添加磁盤的過程，通過這個指令來檢查完成狀況
SQL> select * from v$asm_operation;
19.最後，全部完成的狀态
SQL> select GROUP_NUMBER,DISK_NUMBER,LABEL,NAME,MOUNT_STATUS,state,FAILGROUP,MODE_STATUS,PATH from v$asm_disk where name like '%CD_03_DMORLCEL08';
------------ ----------- ------------------------------- ------------------------------ ------- -------- ------------------------------ -------
4 6 RECO_CD_03_DMORLCEL08 RECO_CD_03_DMORLCEL08 CACHED NORMAL DMORLCEL08 ONLINE o/192.168.10.16/RECO_CD_03_dmorlcel08
查詢griddisk資訊，可以看到asmDiskGroupName，asmDiskName都已經有了。
CellCLI> list griddisk where celldisk=CD_03_dmorlcel08 detail
asmDiskName: DBFS_DG_CD_03_DMORLCEL08
asmDiskGroupName: RECO
asmDiskName: RECO_CD_03_DMORLCEL08
至此，全部的實驗已經完成，如果在更換磁盤後沒有正常添加，可以參考這個實驗。
本文轉自 hsbxxl 51CTO部落格，原文連結：http://blog.51cto.com/hsbxxl/1132925，如需轉載請自行聯系原作者
![oracle中的start with connect by用法[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)