IBM AIX 5L+TSM+Oracle10.2.0.3 RAC+ASM+RMAN经验谈

1. TSM(Tivoli Storage Manager)在产品易用性方面真是不如Veritas的NBU,甚至也不如HP的Omni DP。

2. 高端带库的I/O速度不弱于盘阵,跟I/O通道个数以及驱动器个数关系很大。

3. 使用2个channel往84块单盘300G的底层作了0+1 RAID的21个物理卷组成的ASM磁盘组中(好拗口 …)restore 2T大小的数据库耗时6小时。

4. 目前看来,所有的ASM磁盘组信息都没有存在任何配置文件中,无论是进入asmcmd还是选择v$asm相关视图,都是实时从PV头部读取的信息。所以在ASM使用外部冗余的磁盘组中一块PV坏掉以后,可以直接用dd来清除该磁盘组中所有其他disk的头部信息,然后重新创建磁盘组,然后用RMAN恢复数据库,这也是当ASM磁盘组崩溃以后唯一的修复方法。

5. 条带并不是越多越好。在这个环境中,112块disk在硬件级别做了RAID 0+1的条带和镜像,操作系统中对这112/2*300G=16T的存储做了56个PV,如果在创建LV的时候又选择了条带选项(二次条带化),那么读性能将会严重下降,每秒只能达到60M的读,而如果去除二次条带化,则读性能可以上升到每秒200M。
原因:在将数据写入做了二次条带化的存储上时,首先数据在操作系统级别被打散为56个stripe,然后每个stripe在硬件级别又再次被打散为56个stripe,这样并行写的性能是没有问题的,但是在读取的时候,由于请求的数据在硬件级别是被打散在56块disk中的,而硬件级别的缓存机制在读取一块disk上的数据时将会缓存相邻的大量数据,而这些缓存对于此次读取来说都是无用的,当从另外一个disk中再读取需要的数据时,缓存又需要被腾空再容纳这个disk上的数据,但是这次缓存中又只有很小一部分数据是有用的,因此当PV越多的时候,二次条带将导致越大的性能下降。

6. 10g的新功能change tacking貌似有些bug,在头天晚上启动了change tacking,然后做了level 0备份,2.1T的数据文件总共备份出200G的备份集,第二天做了level 1备份,居然备份出2.1T的备份集,也就是change tracking告诉RMAN在这一天里面所有的data block都发生了变化,所以RMAN备出了数据库中所有block,但是实际上这很明显是不可能的,因为当天的归档日志备份只有500多M,第三天仍旧是level 1备份,就比较正常了,当天产生60多G的归档日志,但是level 1备份只花费了3分钟,这是change tracking真正的威力显示了。

Oracle 10.2.0.3 RAC Reboot due to system time change

在Oracle10.2.0.3 RAC的测试中,发现如果修改某个节点的系统时间超过1.5秒,那么这个节点会被自动重新启动。

好狠的处理方式 ……

详细机制参见Internal Only的Metalink Note 308051.1。

The OPROCD executable sets a signal handler for the SIGALRM handler and sets the interval timer based on the to-millisec parameter provided. The alarm handler gets the current time and checks it against the time that the alarm handler was last entered. If the difference exceeds (to-millisec + margin-millisec), it will fail; the production version will cause a node reboot.

尝试修改/etc/init.cssd中关于OPROCD的配置,将DISABLE_OPROCD设置为TRUE,然后重新启动系统,在系统进程中已经不存在oprocd进程,但是居然修改完系统时间以后,机器仍然被重新启动了。

文档中另外的描述提到,如果OPROCD是在non fatal mode状态下启动的,那么将只会写一段log而不去重新启动机器,并且在Note:265769.1中也描述了如何修改为non fatal mode,但是我没有去尝试。

In fatal mode, OPROCD will reboot the node if it detects excessive wait. In Non Fatal mode, it will write an error message out to the file .oprocd.log in one of the following directories.

最后尝试的结果是将整个cssd进程disable掉,这样可以避免因为修改系统时间而引起机器重启。

Oracle10g的CRS确实有些霸道,上次的测试中拔掉Private IP网卡上的网线,操作系统会重新启动,这次居然修改系统时间也会导致系统重启,真当这些机器是Windows了?UNIX Server中重启一次机器多大的事儿啊,CRS搞的跟吃饭一样随意,时不常reboot。

下面的这段资料描述了Oracle CRS的三个进程会在哪些状态下重新启动机器。

Oracle clusterware has the following three daemons which may be responsible for panicing the node. It is possible that some other external entity may have rebooted the node. In the context of this discussion, we will assume that the reboot/panic was done by an Oracle clusterware daemon.

* Oprocd – Cluster fencing module
* Cssd – Cluster sychronization module which manages node membership
* Oclsomon – Cssd monitor which will monitor for cssd hangs

OPROCD This is a daemon that only gets activated when there is no vendor clusterware present on the OS. This daemon is also not activated to run on Windows/Linux. This daemon runs a tight loop and if it is not scheduled for 1.5 seconds, will reboot the node.
CSSD This daemon pings the other members of the cluster over the private network and Voting disk. If this does not get a response for Misscount seconds and Disktimeout seconds respectively, it will reboot the node.
Oclsomon This daemon monitors the CSSD to ensure that CSSD is scheduled by the OS, if it detects any problems it will reboot the node.

更多讨论参见itpub note 747833。

关于为何要reboot,Wing Hong在上面的帖子里对fencing有一段深入浅出的解释,摘录如下。

fencing is a very important concept in cluster. not sure I can explain clearly in a few words. any how , forget about RAC first, just look at generic share-disk cluster issue.

let’s say the cluster only has two nodes. both share the disk. ie. they both have the right to write to the same disk.

so this share access must be coordinated.

however , if for some reason , this coordination lost, it can be any reason, communication process hanging, you unplug the network cable between them, etc etc

at this moment both nodes are functioning normally except the lost coordination between them.

then the cluster has two problems to solve:
1. who will be the new member of the new incarnation of cluster , in this case, this is split brain issue.

2. once we decide who will remain in the cluster and who will go , how can we prevent the going node NOT to do something harmful to the cluster ? this is fencing issue. Bear in mind , the going node is working perfectly normal except the coordination part, so it still can write to the shared-disk.

There are a couple of approaches in fencing:

1.server fencing , in cluster terms, Shoot The Other Node In The Head (STONITH) , i.e the good node kill the going node.

( the way Oracle do is : reboot itself, once go through the reboot process, it just do the rejoin cluster again, then the cluster can decide whether accept it or not )

2. I/O fencing. rather than trying to kill the node, it is working on the disks side to block the going node’s access to disk, Sun , Veritas has solution on this way.

HTH. also try to do a google on terms like “fencing” , “split brain”, “amnesia”.

Oracle ACE的礼品

一件外套+一个奖杯,DHL送到家里的。

奖杯嘛,普普通通,我原先以为会是黑+红的配色,可惜还是一个大玻璃盘子。
外套倒是着实不错,就是北京这几天倒春寒,要不正好是合适的季节。