Database Notes: April 2017

CRS does not start GIPC error: gipc request failed with 29

CRS does not start GIPC error: gipc request failed with 29

crsd log: getting continuous alert in crsd log file,

2017-04-20 22:58:38.479: [ CSSCLNT][929949424] clssscConnect: gipc request failed with 29 (0x16)
2017-04-20 22:58:38.480: [ CSSCLNT][929949424] clsssInitNative: connect failed, rc 29
2017-04-20 22:58:38.480: [ CRSRTI][929949424] CSS is not ready. Received status 3 from CSS. Waiting for good status ..

cssd log: what I found in the cssd log,

2017-04-20 23:05:28.293: [ GPnP][2588632800]clsgpnpwu_walletfopen: [at clsgpnpwu.c:494] Opened SSO wallet: '/u01/app/grid/product/11.2.0/grid_home/gpnp/alpha/wall

ets/peer/cwallet.sso'

2017-04-20 23:05:28.293: [ GPnP][2588632800]clsgpnp_getCK: [at clsgpnp0.c:1965] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1 of 2 (LSKP-FSW(1))

2017-04-20 23:05:28.293: [ GPnP][2588632800]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>

2017-04-20 23:05:28.293: [ GPnP][2588632800]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=5156, tl=3, f=3

2017-04-20 23:05:28.298: [ CSSD][1131612480]clssscSelect: cookie accept request 0x270f048

2017-04-20 23:05:28.298: [ CSSD][1131612480]clssgmAllocProc: (0x28ec5f0) allocated

2017-04-20 23:05:28.299: [ CSSD][1131612480]clssgmClientConnectMsg: properties of cmProc 0x28ec5f0 - 1,2,3,4

2017-04-20 23:05:28.299: [ CSSD][1131612480]clssgmClientConnectMsg: Connect from con(0x102) proc(0x28ec5f0) pid(5133) version 11:2:1:4, properties: 1,2,3,4

2017-04-20 23:05:28.299: [ CSSD][1131612480]clssgmClientConnectMsg: msg flags 0x0000

2017-04-20 23:05:28.299: [ CSSD][1131612480]clssgmEvtInformation: reqtype (11) cmProc (0x28ec5f0) client ((nil))

2017-04-20 23:05:28.299: [ CSSD][1131612480]clssgmEvtInformation: reqtype (11) req (0x2b130c0)

2017-04-20 23:05:28.299: [ CSSD][1131612480]clssnmQueueNotification: type (11) 0x2b130c0

2017-04-20 23:05:28.313: [GIPCXCPT][2588632800]gipcShutdownF: skipping shutdown, count 3, from [ clsinet.c : 1732], ret gipcretSuccess (0)

2017-04-20 23:05:28.315: [GIPCXCPT][2588632800]gipcShutdownF: skipping shutdown, count 2, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)

2017-04-20 23:05:28.315: [GIPCGMOD][2588632800]gipcmodGipcPassInitializeNetwork: using host information 192.168.190.1

2017-04-20 23:05:28.316: [ CSSD][2588632800]clssnmOpenGIPCEndp: listening on gipc://alpha:nm_rac-cluster#192.168.190.1#20249

2017-04-20 23:05:28.316: [ CSSD][2588632800]clssnmInitNMInfo: Initializing uniqueness 0

2017-04-20 23:05:28.316: [ CSSD][2588632800]clssnmReadDiscoveryProfile: voting file discovery string()

2017-04-20 23:05:28.316: [ CSSD][2588632800]clssnkInit: NK generic layer initializing.

2017-04-20 23:05:28.316: [ CSSD][2588632800]clssscGetParameterOLR: OLR fetch for parameter GIPC NM trclvl (12) failed with rc 21

2017-04-20 23:05:28.327: [ CLSF][1140005184]Allocated CLSF context

2017-04-20 23:05:28.327: [ CSSD][1140005184]clssnmvDDiscThread: using discovery string for initial discovery

2017-04-20 23:05:28.327: [ SKGFD][1140005184]Discovery with str::

2017-04-20 23:05:28.327: [ SKGFD][1140005184]UFS discovery with ::

2017-04-20 23:05:28.327: [ SKGFD][1140005184]OSS discovery with ::

2017-04-20 23:05:28.328: [ SKGFD][1140005184]Discovery with asmlib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: str ::

2017-04-20 23:05:28.328: [ CSSD][1140005184]clssnmvDiskVerify: Successful discovery of 0 disks

2017-04-20 23:05:28.328: [ CSSD][1140005184]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery

2017-04-20 23:05:28.328: [ CSSD][1140005184]clssnmvFindInitialConfigs: No voting files found

2017-04-20 23:05:28.328: [ CSSD][1140005184]###################################

2017-04-20 23:05:28.328: [ CSSD][1140005184]clssscExit: CSSD signal 11 in thread clssnmvDDiscThread

The successful start of CRSD depends on the discovery of the OCR and a fully functional CSS daemon. As with CSSD, a functional network and GIPC daemon are required for crsd.bin to work properly. If ocssd.bin has not been created, then the log file will show above errors:

To resolve this, check whether shared storage is accessible or not, as VOTING DISK & OCR files resides in shared storage (ASM).

# /etc/init.d/oracleasm listdisks

2) Previous in my environment, /etc/sysconfig/oracleasm file contains below entries:

ORACLEASM_SCANORDER=”dm”

ORACLEASM_SCANEXCLUDE=”sd”

So, I have simply removed pair value to defualt,

ORACLEASM_SCANORDER=” ”

ORACLEASM_SCANEXCLUDE=” ”

3) restart the asmlib by (except 1st node):

# /etc/init.d/oracleasm restart

4) Scan ASM disks:

# /etc/init.d/oracleasm scandisks

Scanning the system for Oracle ASMLib disks: [ OK ]

8) List the ASM disks:

# /etc/init.d/oracleasm listdisks

DATA1

9) Now restart the CRS

# crsctl stop crs -f

# crsctl start crs

# crsctl check cluster -all

**************************************************************

alpha:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

beta:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

Creation of shared storage for ASM (Automatic Shared Storage) to be used for RAC

Creation of shared storage for ASM (Automatic Shared Storage) to be used for RAC

1) add 1 more hard disk to existing setup, of fixed size 30GB.

2) Verifty whether is it available or not

# fdisk -l

3) Create partition

# fdisk /dev/sdb

Command (m for help): n
Command action
e extended
p primary partition (1-4)

p

Partition number (1-4): 1
First cylinder (1-3916, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-3916, default 3916):
Using default value 3916

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

# fdisk -l

Disk /dev/sda: 85.8 GB, 85899345920 bytes
255 heads, 63 sectors/track, 10443 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 1530 12289693+ 83 Linux
/dev/sda2 1531 6629 40957717+ 83 Linux
/dev/sda3 6630 7904 10241437+ 83 Linux
/dev/sda4 7905 10443 20394517+ 5 Extended
/dev/sda5 7905 8796 7164958+ 82 Linux swap / Solaris
/dev/sda6 8797 9561 6144831 83 Linux
/dev/sda7 9562 10198 5116671 83 Linux

Disk /dev/sdb: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 3916 31455238+ 83 Linux

# cd /dev/
# ls sd*
sda sda1 sda2 sda3 sda4 sda5 sda6 sda7 sdb sdb1

3) Change the ownership of created partion

# chown grid:oinstall /dev/sdb1
# chown grid:oinstall /dev/sdb
# chmod 660 /dev/sdb1
# chmod 660 /dev/sdb

4) Configure asm

# /etc/init.d/oracleasm configure -i

Default user to own the driver interface []: grid
Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]

5) Create label to partition

# /etc/init.d/oracleasm createdisk data1 /dev/sdb1
Marking disk "data1" as an ASM disk: [ OK ]

6)

# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]

7)

# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]

8)

# /etc/init.d/oracleasm listdisks
DATA1

MIGRATION FROM NON ASM TO ASM DATABASE (11g)

---- MIGRATION FROM NON ASM TO ASM DATABASE (11g)----

sql> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

sql> select file_name from dba_data_files;

FILE_NAME
--------------------------------------------------------------------------------
/u01/kings/king/users01.dbf
/u01/kings/king/undotbs01.dbf
/u01/kings/king/sysaux01.dbf
/u01/kings/king/system01.dbf

sql> select file_name from dba_temp_files;

FILE_NAME
--------------------------------------------------------------------------------
/u01/kings/king/temp01.dbf

sql> show parameter spfile;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string /u01/app/oracle/product/1120/d
b_home/dbs/spfileking.ora

sql> show parameter control_file;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_file_record_keep_time integer 7
control_files string /u01/kings/king/control01.ctl,
/u01/app/oracle/flash_recover
y_area/king/control02.ctl

sql> alter system set control_files='+DATA' scope=spfile;
System altered.

sql> alter system set db_create_file_dest='+DATA' scope=spfile;
System altered.

sql> shutdown immediate;

sql> startup nomount;

$ rman target /

Recovery Manager: Release 11.2.0.1.0 - Production on Sun Apr 23 19:14:34 2017

Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.

connected to target database: KING (not mounted)

RMAN> restore controlfile from '/u01/kings/king/control01.ctl';

Starting restore at 23-APR-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=23 device type=DISK

channel ORA_DISK_1: copied control file copy
output file name=+DATA/king/controlfile/current.256.942088523
Finished restore at 23-APR-17

RMAN> alter database mount;

database mounted
released channel: ORA_DISK_1

RMAN> backup as copy database format '+DATA';

Starting backup at 23-APR-17
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=23 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=/u01/kings/king/system01.dbf
output file name=+DATA/king/datafile/system.257.942088645 tag=TAG20170423T191723 RECID=1 STAMP=942088688
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:45
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=/u01/kings/king/sysaux01.dbf
output file name=+DATA/king/datafile/sysaux.258.942088689 tag=TAG20170423T191723 RECID=2 STAMP=942088720
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:35
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=/u01/kings/king/undotbs01.dbf
output file name=+DATA/king/datafile/undotbs1.259.942088725 tag=TAG20170423T191723 RECID=3 STAMP=942088727
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
copying current control file
output file name=+DATA/king/controlfile/backup.260.942088731 tag=TAG20170423T191723 RECID=4 STAMP=942088734
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=/u01/kings/king/users01.dbf
output file name=+DATA/king/datafile/users.261.942088735 tag=TAG20170423T191723 RECID=5 STAMP=942088735
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 23-APR-17
channel ORA_DISK_1: finished piece 1 at 23-APR-17
piece handle=+DATA/king/backupset/2017_04_23/nnsnf0_tag20170423t191723_0.262.942088735 tag=TAG20170423T191723 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 23-APR-17

RMAN> switch database to copy;

datafile 1 switched to datafile copy "+DATA/king/datafile/system.257.942088645"
datafile 2 switched to datafile copy "+DATA/king/datafile/sysaux.258.942088689"
datafile 3 switched to datafile copy "+DATA/king/datafile/undotbs1.259.942088725"
datafile 4 switched to datafile copy "+DATA/king/datafile/users.261.942088735"

RMAN> ALTER DATABASE OPEN;

database opened

RMAN> EXIT

Recovery Manager complete.

$ sqlplus / as sysdba

SQL> select file_name from dba_data_files;

FILE_NAME
--------------------------------------------------------------------------------
+DATA/king/datafile/users.261.942088735
+DATA/king/datafile/undotbs1.259.942088725
+DATA/king/datafile/sysaux.258.942088689
+DATA/king/datafile/system.257.942088645

SQL> select file_name from dba_temp_files;

FILE_NAME
--------------------------------------------------------------------------------
/u01/kings/king/temp01.dbf

SQL> alter database tempfile '/u01/kings/king/temp01.dbf' drop including datafiles;

Database altered.

SQL> select file_name from dba_temp_files;

no rows selected

SQL> alter tablespace temp add tempfile size 300M;

Tablespace altered.

SQL> select file_name from dba_temp_files;

FILE_NAME
--------------------------------------------------------------------------------
+DATA/king/tempfile/temp.263.942088985

SQL> select member from v$logfile;

MEMBER
--------------------------------------------------------------------------------
/u01/kings/king/redo03.log
/u01/kings/king/redo02.log
/u01/kings/king/redo01.log

SQL> alter database add logfile group 4 '+DATA' size 10M;
alter database add logfile group 5 '+DATA' size 10M;
alter database add logfile group 6 '+DATA' size 10M;

Database altered.

SQL>
Database altered.

SQL>

Database altered.

SQL> SQL>
SQL>
SQL> select member from v$logfile;

MEMBER
--------------------------------------------------------------------------------
/u01/kings/king/redo03.log
/u01/kings/king/redo02.log
/u01/kings/king/redo01.log
+DATA/king/onlinelog/group_4.264.942089039
+DATA/king/onlinelog/group_5.265.942089041
+DATA/king/onlinelog/group_6.266.942089041

6 rows selected.

SQL> select group#,status from v$log;

GROUP# STATUS
---------- ----------------
1 INACTIVE
2 CURRENT
3 UNUSED
4 UNUSED
5 UNUSED
6 UNUSED

6 rows selected.

SQL> alter database drop logfile group 3;

Database altered.

SQL> select group#,status from v$log;

GROUP# STATUS
---------- ----------------
1 INACTIVE
2 CURRENT
4 UNUSED
5 UNUSED
6 UNUSED

SQL> alter database drop logfile group 1;

Database altered.

SQL> select group#,status from v$log;

GROUP# STATUS
---------- ----------------
2 CURRENT
4 UNUSED
5 UNUSED
6 UNUSED

SQL> alter system switch logfile;

System altered.

SQL> select group#,status from v$log;

GROUP# STATUS
---------- ----------------
2 ACTIVE
4 CURRENT
5 UNUSED
6 UNUSED

SQL> select group#,status from v$log;

GROUP# STATUS
---------- ----------------
2 ACTIVE
4 CURRENT
5 UNUSED
6 UNUSED

SQL> alter system checkpoint;

System altered.

SQL> select group#,status from v$log;

GROUP# STATUS
---------- ----------------
2 INACTIVE
4 CURRENT
5 UNUSED
6 UNUSED

SQL> alter database drop logfile group 2;

Database altered.

SQL>

Tuesday 25 April 2017

CRS does not start GIPC error: gipc request failed with 29

Creation of shared storage for ASM (Automatic Shared Storage) to be used for RAC

MIGRATION FROM NON ASM TO ASM DATABASE (11g)