由于RAC的两个节点要进行时间同步。目前采用的方法是一个节点从另一个节点直接同步时间。结果发现RAC环境中的同步时间的那个节点上的Oracle实例重启了。

前一段时间看了Kamus在itpub上的帖子：http://www.itpub.net/showthread.php?s=&threadid=747833。还特意测试了一下，没有发现类似的问题。

结果没过多长时间，自己就碰到了。不过虽然情况类似，但是没有那么野蛮，只是将节点二的数据库重启了，而没有重启系统。而且，数据库关闭是采用的还是正常的关闭方式。

摘录一些log如下，节点1上的alert文件：

Thu May 24 21:15:37 2007
Reconfiguration started (old inc 4, new inc 6)
List of nodes:
0
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Thu May 24 21:15:37 2007
LMS 0: 0 GCS shadows cancelled, 0 closed
Thu May 24 21:15:37 2007
LMS 1: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Thu May 24 21:15:38 2007
Instance recovery: looking for dead threads
Thu May 24 21:15:38 2007
Beginning instance recovery of 1 threads
Thu May 24 21:15:39 2007
LMS 0: 453637 GCS shadows traversed, 0 replayed
Thu May 24 21:15:39 2007
LMS 1: 456482 GCS shadows traversed, 0 replayed
Thu May 24 21:15:39 2007
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
Thu May 24 21:15:41 2007
parallel recovery started with 7 processes
Thu May 24 21:15:41 2007
Started redo scan
Thu May 24 21:15:42 2007
Completed redo scan
182 redo blocks read, 27 data blocks need recovery
Thu May 24 21:15:42 2007
Started redo application at
Thread 2: logseq 52, block 379401
Thu May 24 21:15:42 2007
Recovery of Online Redo Log: Thread 2 Group 8 Seq 52 Reading mem 0
Mem# 0: /dev/vx/rdsk/datadg/tradedb_redo2_4_1_1g
Mem# 1: /dev/vx/rdsk/datadg/tradedb_redo2_4_2_1g
Thu May 24 21:15:42 2007
Completed redo application
Thu May 24 21:15:42 2007
Completed instance recovery at
Thread 2: logseq 52, block 379583, scn 5137925732
27 data blocks read, 27 data blocks written, 182 redo blocks read
Switch log for thread 2 to sequence 53
Thu May 24 21:21:16 2007
Reconfiguration started (old inc 6, new inc 8)
List of nodes:
0 1
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Thu May 24 21:21:16 2007
LMS 0: 0 GCS shadows cancelled, 0 closed
Thu May 24 21:21:16 2007
LMS 1: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Thu May 24 21:21:17 2007
LMS 0: 11529 GCS shadows traversed, 4001 replayed
Thu May 24 21:21:17 2007
LMS 1: 11363 GCS shadows traversed, 4001 replayed
LMS 1: 11403 GCS shadows traversed, 4001 replayed
Thu May 24 21:21:17 2007
LMS 0: 11539 GCS shadows traversed, 4001 replayed
LMS 0: 11294 GCS shadows traversed, 4001 replayed
Thu May 24 21:21:17 2007
LMS 1: 11492 GCS shadows traversed, 4001 replayed