Tag: Oracle11gR2

Install 11.2.0.2 RAC on OEL5.5 x86-64 (root.sh issue on second node)

2 Comments

在安装11.2.0.2 RAC的时候，第一步安装Grid，在第二个节点上运行root.sh的时候，报错如下：

Start of resource "ora.ctssd" failed
CRS-2672: Attempting to start 'ora.ctssd' on 'xsh-server2'
CRS-2674: Start of 'ora.ctssd' on 'xsh-server2' failed
CRS-4000: Command Start failed, or completed with errors.
Cluster Time Synchronisation Service  start in exclusive mode failed at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 6455.
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

从报错信息上看是ctssd进程启动失败（在这之前会显示cssd进程启动成功，这与MOS上的其它一些第二节点运行root.sh失败的情形是不一样的，那些场景在cssd进程启动的时候就失败了），查看ctssd进程的启动log（位于$GRID_HOME/log/ctssd目录下），发现如下错误信息。

2010-11-12 18:55:46.132: [    GIPC][2424495392] gipcCheckInitialization: possible incompatible non-threaded init from [prom.c : 687], original from [clsss.c : 5325]
[ default][2424495392]Failure 4 in trying to open SV key SYSTEM.version.localhost

[ default][2424495392]procr_open_key error 4 errorbuf : PROCL-4: The local registry key to be operated on does not exist.

2010-11-12 18:55:46.135: [    CTSS][2424495392]clsctss_r_av2: Error [3] retrieving Active Version from OLR. Returns [19].
2010-11-12 18:55:46.138: [    CTSS][2424495392](:ctss_init16:): Error [19] retrieving active version. Returns [19].
2010-11-12 18:55:46.138: [    CTSS][2424495392]ctss_main: CTSS init failed [19]
2010-11-12 18:55:46.138: [    CTSS][2424495392]ctss_main: CTSS daemon aborting [19].
2010-11-12 18:55:46.138: [    CTSS][2424495392]CTSS daemon aborting

从crsctl命令中也可以看出ora.cssd启动成功，但是ora.ctssd是OFFLINE状态。

 $ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        OFFLINE OFFLINE                                                   
ora.cluster_interconnect.haip
      1        OFFLINE OFFLINE                                                   
ora.crf
      1        OFFLINE OFFLINE                                                   
ora.crsd
      1        OFFLINE OFFLINE                                                   
ora.cssd
      1        ONLINE  ONLINE       xsh-server2                                  
ora.cssdmonitor
      1        ONLINE  ONLINE       xsh-server2                                  
ora.ctssd
      1        ONLINE  OFFLINE                                                   
ora.diskmon
      1        ONLINE  ONLINE       xsh-server2                                  
ora.drivers.acfs
      1        OFFLINE OFFLINE                                                   
ora.evmd
      1        OFFLINE OFFLINE                                                   
ora.gipcd
      1        ONLINE  ONLINE       xsh-server2                                  
ora.gpnpd
      1        ONLINE  ONLINE       xsh-server2                                  
ora.mdnsd
      1        ONLINE  ONLINE       xsh-server2

此时如果用此命令查看第一个节点的状况会发现所有资源都是正常ONLINE的。继续检查cssd.log（位于$GRID_HOME/log/cssd目录中），显示在发现ASM磁盘的时候报错。

2010-11-12 13:44:30.505: [   SKGFD][1087203648]UFS discovery with :ORCL:VOL*:

2010-11-12 13:44:30.505: [   SKGFD][1087203648]OSS discovery with :ORCL:VOL*:

2010-11-12 13:44:30.505: [   SKGFD][1087203648]Discovery with asmlib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: str :ORCL:VOL*:

2010-11-12 13:44:30.505: [   SKGFD][1087203648]Fetching asmlib disk :ORCL:VOL1:

2010-11-12 13:44:30.505: [   SKGFD][1087203648]Fetching asmlib disk :ORCL:VOL2:

2010-11-12 13:44:30.505: [   SKGFD][1087203648]Fetching asmlib disk :ORCL:VOL3:

2010-11-12 13:44:30.505: [   SKGFD][1087203648]ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op asm_open error Operation not permitted
)
2010-11-12 13:44:30.505: [   SKGFD][1087203648]ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op asm_open error Operation not permitted
)
2010-11-12 13:44:30.505: [   SKGFD][1087203648]ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op asm_open error Operation not permitted

值得注意的是，这样的报错在第一个节点上也同样存在，但是第一个节点上所有的资源包括ASM磁盘组却都是正常运行的。

对于以上cssd.log中的错误，按照MOS Note [1050164.1]处理，修改/etc/sysconfig/oracleasm-_dev_oracleasm文件，指定ASMLib在发现磁盘的时候需要忽略的盘和需要检查的盘。在我们的环境中是使用了Multipath来对多块磁盘做多路径处理，因此需要包括dm开头的磁盘，而忽略sd开头的磁盘。这样的问题也应该只会发生在使用了Multipath的磁盘上。

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="dm"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"

可以通过以下方法来确认是否遭遇了此问题。

# ls -l /dev/oracleasm/disks
brw-rw---- 1 oracle dba 3, 65 May 14 12:08 CRSVOL
# cat /proc/partitions
  3 65 4974448 sda
253  1 4974448 dm-1

在上面可以看到CRSVOL这个用oracleasm创建的ASM磁盘的major和minor号分别是3，65，而这正是/dev/sda的号，并不是/dev/dm-1的号，所以表示在创建ASM磁盘组的时候并没有使用到Multipath设备。通常情况下，在节点1上是正确的，而在节点2上不正确的，因此出现了问题。

在处理完以上问题以后，必须要对grid环境做deconfig再reconfig，而不能只是在失败节点上重新运行root.sh（我在这里耗费了大量时间），重新配置grid的步骤可以参考MOS Note [942166.1] – How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation。之后root.sh顺利在第二节点上运行成功。

在错误解决以后，回顾之前的安装信息，可以发现虽然第一个节点显示所有资源都正常，但是和正常的root.sh运行信息相比则缺少了几行显示。

正常的信息如下：

# $GRID_HOME/root.sh
Running Oracle 11g root script...

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE 
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
OLR initialization - successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-2672: Attempting to start 'ora.mdnsd' on 'xsh-server1'
CRS-2676: Start of 'ora.mdnsd' on 'xsh-server1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'xsh-server1'
CRS-2676: Start of 'ora.gpnpd' on 'xsh-server1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'xsh-server1'
CRS-2672: Attempting to start 'ora.gipcd' on 'xsh-server1'
CRS-2676: Start of 'ora.cssdmonitor' on 'xsh-server1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'xsh-server1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'xsh-server1'
CRS-2672: Attempting to start 'ora.diskmon' on 'xsh-server1'
CRS-2676: Start of 'ora.diskmon' on 'xsh-server1' succeeded
CRS-2676: Start of 'ora.cssd' on 'xsh-server1' succeeded

ASM created and started successfully.

Disk Group CRSDG created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Successful addition of voting disk 67463e71af084f76bf98b3ee55081e40.
Successfully replaced voting disk group with +CRSDG.
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   67463e71af084f76bf98b3ee55081e40 (ORCL:VOL1) [CRSDG]
Located 1 voting disk(s).

CRS-2672: Attempting to start 'ora.asm' on 'xsh-server1'
CRS-2676: Start of 'ora.asm' on 'xsh-server1' succeeded
CRS-2672: Attempting to start 'ora.CRSDG.dg' on 'xsh-server1'
CRS-2676: Start of 'ora.CRSDG.dg' on 'xsh-server1' succeeded
ACFS-9200: Supported
ACFS-9200: Supported
CRS-2672: Attempting to start 'ora.registry.acfs' on 'xsh-server1'
CRS-2676: Start of 'ora.registry.acfs' on 'xsh-server1' succeeded
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

而之前的信息则缺少了以下4行。

LOCAL ADD MODE 
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
OLR initialization - successful

Oracle显然不会承认这是bug，好吧，解决问题就好。

11.2.0.2 Patch Set? Really a Patch Set?

6 Comments

Oracle最新为Linux x86和x86-64平台发布了Oracle 11.2.0.2 patch set [Patch set ID: 10098816]，这个庞大的patch set拥有5G的超大容量。所以几乎全世界的人都在问，这真的是一个补丁集吗？不仅仅是体积庞大，甚至还一改往日补丁集仅仅是作为Bug修复的角色，在11.2.0.2中还包含了很多New Feature。

Oracle专门为此补丁写了一篇文章-Important Changes to Oracle Database Patch Sets Starting With 11.2.0.2 [ID 1189783.1]，着重提到了几点。

1. Oracle 11.2.0.2 patch set是一个完整的安装包，以后Oracle也会如此发布后续的补丁包。所以在以后如果要安装新数据软件，不再需要安装一个Base版本然后再打补丁集到最新版本，而是直接使用最新补丁集安装包即可。
2. Oracle建议今后的补丁集安装都安装在一个与当前环境目录不同的全新目录下(Out-of-place upgrade)，在安装完成以后再通过此份软件启动数据库实例。实际上以后安装补丁集的过程就等同于从一个版本升级到一个新版本的过程。
3. 为什么做这种改变，是因为Oracle认为Out-of-place upgrade是最佳实践，比较安全。当然这实际上符合Oracle的作风，他认为合理且最好的通常都会武断地改变并且说服客户去接受。
4. Oracle仅仅计划在11.2.0.2中提供一些新功能，今后的补丁集仍然会以修复bug为首要任务。

我个人承认确实Out-of-place upgrade是更加安全的，但是至少这种做法目前存在两点缺陷：
1. 补丁尺寸急剧增加，下载2G的补丁和下载5G的补丁还是有很大区别，但是这不是大问题。当然，由于需要安装在新Oracle Home中，在打补丁时需要额外的硬盘空间，但是不到10G的空间需求也不是大问题。
2. 由于在打完补丁以后，Oracle Home路径实际上发生了变化，那么在脚本中，服务定义中牵涉到的原ORACLE_HOME路径必须都要做相应改动。这是个大问题。不过，可以将老目录rename成一个新名字，再将新目录rename成原目录。

今后的补丁集发布计划可以参看Release Schedule of Current Database Releases [ID 742060.1]

How to create ASM filesystem(ACFS) in Oracle 11gR2

2 Comments

本文描述如何创建ACFS文件系统，前提要求：
1. Oracle 11gR2数据库。
2. 如果是11.2.0.1数据库那么操作系统只支持Linux和Windows，如果是11.2.0.2数据库那么增加支持的操作系统有AIX和Solaris。
3. 已经安装Grid Infrastructure，单机即可（但是ACFS在Oracle Restart环境中会有些限制，详见【备注1】）。
4. 已经创建了ASM实例以及ASM磁盘组，实例中ASM实例名为+ASM，磁盘组为ORADG。

[sourcecode language=”sql”]SQL> alter diskgroup oradg add volume acfsvol size 1G;
alter diskgroup oradg add volume acfsvol size 1G
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15221: ASM operation requires compatible.asm of 11.2.0.0.0 or higher[/sourcecode]

如上的错误表示如果要在ASM磁盘组上创建ACFS Volume，必须要求ASM磁盘组的属性COMPATIBLE.ASM在11.2以上。
如果ASM磁盘组是使用asmca图形化工具创建的，那么compatible.asm默认设置就已经为11.2，但如果是使用CREATE DISKGROUP这个SQL命令创建的，那么默认设置则为10.1，需要手动修改。

[sourcecode language=”sql” toolbar=”false”]SQL> alter diskgroup oradg set attribute ‘COMPATIBLE.ASM’=’11.2’;[/sourcecode]

如果要创建ACFS Volume，还必须要求ASM磁盘组的COMPATIBLE.ADVM属性也在11.2以上，此属性默认为空。

[sourcecode language=”sql”]SQL> alter diskgroup oradg set attribute ‘compatible.advm’=’11.2’;
alter diskgroup oradg set attribute ‘compatible.advm’=’11.2’
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15242: could not set attribute compatible.advm
ORA-15238: 11.2 is not a valid value for attribute compatible.advm
ORA-15477: cannot communicate with the volume driver

SQL> host oerr ora 15477
15477, 00000, "cannot communicate with the volume driver"
// *Cause: An attempt was made to communicate with the volume driver.
// *Action: Check that the ASM volume driver is loaded. If so, check the alert
// log to identify the reason for failure and take necessary action
// to prevent such failures in the future.
//[/sourcecode]

如上的错误表示ASM volume driver没有加载。需要使用root用户手工加载。

# /app/grid/bin/acfsload start -s

然后再次修改ASM磁盘组的COMPATIBLE.ADVM属性，并创建ACFS Volume。

[sourcecode language=”sql”]SQL> alter diskgroup oradg set attribute ‘compatible.advm’=’11.2′;

Diskgroup altered.

SQL> alter diskgroup oradg add volume acfsvol size 1G;

Diskgroup altered.

SQL> select VOLUME_DEVICE from V$ASM_VOLUME where VOLUME_NAME=’ACFSVOL’;

VOLUME_DEVICE
——————————————————————————–
/dev/asm/acfsvol-351[/sourcecode]

创建并注册文件系统，然后使用mount.acfs命令挂载文件系统，以下命令需要用root用户执行。

# mkdir -p /app/oracle/acfsmounts/oradg_acfsvol 
# /sbin/mkfs -t acfs -n ACFSVOL1 /dev/asm/acfsvol-351
mkfs.acfs: version                   = 11.2.0.1.0.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/acfsvol-351
mkfs.acfs: volume size               = 1073741824
mkfs.acfs: Format complete.
# /sbin/acfsutil registry -a -f /dev/asm/acfsvol-351 /app/oracle/acfsmounts/oradg_acfsvol
acfsutil registry: mount point /app/oracle/acfsmounts/oradg_acfsvol successfully added to Oracle Registry
# mount.acfs -o all
#df -k | grep acfs
/dev/asm/acfsvol-351   1048576     39192   1009384   4% /app/oracle/acfsmounts/oradg_acfsvol
# chown oracle:dba /app/oracle/acfsmounts/oradg_acfsvol

创建成功以后，可以用oracle用户在该文件系统下创建测试文件。

$ dd if=/dev/zero of=/app/oracle/acfsmounts/oradg_acfsvol/testfile bs=8192 count=100
100+0 records in
100+0 records out
819200 bytes (819 kB) copied, 0.0270009 seconds, 30.3 MB/s
$ ls -l /app/oracle/acfsmounts/oradg_acfsvol/testfile
-rw-r--r-- 1 oracle dba 819200 Aug 12 19:18 /app/oracle/acfsmounts/oradg_acfsvol/testfile

查看ACFS文件系统信息。

# /sbin/acfsutil info fs 
/app/oracle/acfsmounts/oradg_acfsvol
    ACFS Version: 11.2.0.1.0.0
    flags:        MountPoint,Available
    mount time:   Thu Aug 12 19:06:33 2010
    volumes:      1
    total size:   1073741824
    total free:   968667136
    primary volume: /dev/asm/acfsvol-351
        label:                 ACFSVOL1
        flags:                 Primary,Available,ADVM
        on-disk version:       39.0
        allocation unit:       4096
        major, minor:          252, 179713
        size:                  1073741824
        free:                  968667136
        ADVM diskgroup         ORADG
        ADVM resize increment: 268435456
        ADVM redundancy:       unprotected
        ADVM stripe columns:   4
        ADVM stripe width:     131072
    number of snapshots:  0
    snapshot space usage: 0

在ASM卷被打开的时候，无法直接shutdown ASM实例，会报ORA-15487错误。

[sourcecode language=”sql” toolbar=”false”]SQL> shutdown immediate
ORA-15487: cannot shutdown the ASM instance with an open ASM volume[/sourcecode]

可以使用umount命令卸载ACFS文件系统。

# /bin/umount -t acfs -a

如果挂载文件系统时报错，那么可能是因为ACFS Volume没有激活，Volume的状态可以从V$ASM_VOLUME.STATE字段获得，显示为“ENABLED”才表示已激活。

# mount.acfs /dev/asm/acfsvol-351 /app/oracle/acfsmounts/oradg_acfsvol
mount.acfs: CLSU-00100: Operating System function: open64 (/dev/asm/acfsvol-351) failed with error data: 2
mount.acfs: CLSU-00101: Operating System error message: No such file or directory
mount.acfs: CLSU-00103: error location: OOF_1
mount.acfs: ACFS-02017: Failed to open volume /dev/asm/acfsvol-351. Verify the volume exists.

如果Volume状态显示为DISABLE，可以使用如下命令，激活Volume。
[sourcecode language=”sql” toolbar=”false”]SQL>alter diskgroup ORADG enable volume ‘ACFSVOL’;[/sourcecode]

本文创建使用的是SQL命令行方式创建ACFS卷，用asmcmd也可以完成，可以参看官方文档 | Surachart的文章。

而如果选用图像化界面的话，可以用asmca或者OEM来完成，下图是asmca界面，在其中查看命令行写法也很方便。

【备注1】
在当前版本的Oracle Restart环境（也就是Standalone Grid Infrastructure）中，以下操作不会自动运行。
1. 加载Oracle ACFS drivers
2. 加载存在于ACFS mount registry中的Oracle ACFS文件系统

应对于第一个问题，我们可以通过以下方法让操作系统在启动的时候自动加载Oracle ACFS drivers。

创建/etc/init.d/acfsload文件，让其在操作系统启动时自动运行。

# cat /etc/init.d/acfsload
#!/bin/sh
# chkconfig: 2345 30 21
# description: Load Oracle ACFS drivers at system boot
/app/grid/bin/acfsload start -s

# chmod u+x /etc/init.d/acfsload
# chkconfig --add  acfsload
# chkconfig --list  acfsload   
acfsload        0:off   1:off   2:on    3:on    4:on    5:on    6:off

对于第二个问题，由于ACFS文件系统能够正确加载，必须要求ASM实例启动成功，并且相应的ASM磁盘组正确加载，这份依赖性在集群环境中是通过创建Oracle ACFS registry resource (ora.registry.acfs)来实现的，但是在Standalone环境中，我们无法保证这份依赖性，因此只能通过创建以下脚本用root用户手动挂载ACFS文件系统（如果你们有更好的方法请告诉我）。

# cat /usr/local/sbin/mountacfs 
su - grid -c "asmcmd volenable -G oradg -a"
mount.acfs -o all