CRS does not start GIPC error: gipc request failed with 29

CRS does not start GIPC error: gipc request failed with 29


crsd log: getting continuous alert in crsd log file, 

2017-04-20 22:58:38.479: [ CSSCLNT][929949424] clssscConnect: gipc request failed with 29 (0x16)
2017-04-20 22:58:38.480: [ CSSCLNT][929949424] clsssInitNative: connect failed, rc 29
2017-04-20 22:58:38.480: [  CRSRTI][929949424] CSS is not ready. Received status 3 from CSS. Waiting for good status ..

cssd log: what I found in the cssd log,

2017-04-20 23:05:28.293: [    GPnP][2588632800]clsgpnpwu_walletfopen: [at clsgpnpwu.c:494] Opened SSO wallet: '/u01/app/grid/product/11.2.0/grid_home/gpnp/alpha/wall

ets/peer/cwallet.sso'

2017-04-20 23:05:28.293: [    GPnP][2588632800]clsgpnp_getCK: [at clsgpnp0.c:1965] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1 of 2 (LSKP-FSW(1))

2017-04-20 23:05:28.293: [    GPnP][2588632800]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>

2017-04-20 23:05:28.293: [    GPnP][2588632800]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=5156, tl=3, f=3

2017-04-20 23:05:28.298: [    CSSD][1131612480]clssscSelect: cookie accept request 0x270f048

2017-04-20 23:05:28.298: [    CSSD][1131612480]clssgmAllocProc: (0x28ec5f0) allocated

2017-04-20 23:05:28.299: [    CSSD][1131612480]clssgmClientConnectMsg: properties of cmProc 0x28ec5f0 - 1,2,3,4

2017-04-20 23:05:28.299: [    CSSD][1131612480]clssgmClientConnectMsg: Connect from con(0x102) proc(0x28ec5f0) pid(5133) version 11:2:1:4, properties: 1,2,3,4

2017-04-20 23:05:28.299: [    CSSD][1131612480]clssgmClientConnectMsg: msg flags 0x0000

2017-04-20 23:05:28.299: [    CSSD][1131612480]clssgmEvtInformation: reqtype (11) cmProc (0x28ec5f0) client ((nil))

2017-04-20 23:05:28.299: [    CSSD][1131612480]clssgmEvtInformation: reqtype (11) req (0x2b130c0)

2017-04-20 23:05:28.299: [    CSSD][1131612480]clssnmQueueNotification: type (11) 0x2b130c0

2017-04-20 23:05:28.313: [GIPCXCPT][2588632800]gipcShutdownF: skipping shutdown, count 3, from [ clsinet.c : 1732], ret gipcretSuccess (0)

2017-04-20 23:05:28.315: [GIPCXCPT][2588632800]gipcShutdownF: skipping shutdown, count 2, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)

2017-04-20 23:05:28.315: [GIPCGMOD][2588632800]gipcmodGipcPassInitializeNetwork: using host information 192.168.190.1

2017-04-20 23:05:28.316: [    CSSD][2588632800]clssnmOpenGIPCEndp: listening on gipc://alpha:nm_rac-cluster#192.168.190.1#20249

2017-04-20 23:05:28.316: [    CSSD][2588632800]clssnmInitNMInfo: Initializing uniqueness 0

2017-04-20 23:05:28.316: [    CSSD][2588632800]clssnmReadDiscoveryProfile: voting file discovery string()

2017-04-20 23:05:28.316: [    CSSD][2588632800]clssnkInit: NK generic layer initializing.

2017-04-20 23:05:28.316: [    CSSD][2588632800]clssscGetParameterOLR: OLR fetch for parameter GIPC NM trclvl (12) failed with rc 21

2017-04-20 23:05:28.327: [    CLSF][1140005184]Allocated CLSF context

2017-04-20 23:05:28.327: [    CSSD][1140005184]clssnmvDDiscThread: using discovery string  for initial discovery

2017-04-20 23:05:28.327: [   SKGFD][1140005184]Discovery with str::

2017-04-20 23:05:28.327: [   SKGFD][1140005184]UFS discovery with ::

2017-04-20 23:05:28.327: [   SKGFD][1140005184]OSS discovery with ::

2017-04-20 23:05:28.328: [   SKGFD][1140005184]Discovery with asmlib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: str ::

2017-04-20 23:05:28.328: [    CSSD][1140005184]clssnmvDiskVerify: Successful discovery of 0 disks

2017-04-20 23:05:28.328: [    CSSD][1140005184]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery

2017-04-20 23:05:28.328: [    CSSD][1140005184]clssnmvFindInitialConfigs: No voting files found

2017-04-20 23:05:28.328: [    CSSD][1140005184]###################################

2017-04-20 23:05:28.328: [    CSSD][1140005184]clssscExit: CSSD signal 11 in thread clssnmvDDiscThread

The successful start of CRSD depends on the discovery of the OCR and a fully functional CSS daemon. As with CSSD, a functional network and GIPC daemon are required for crsd.bin to work properly. If ocssd.bin has not been created, then the log file will show above errors:

To resolve this, check whether shared storage is accessible or not, as VOTING DISK & OCR files resides in shared storage (ASM).

1)

# /etc/init.d/oracleasm listdisks

#

2) Previous in my environment, /etc/sysconfig/oracleasm file contains below entries:

ORACLEASM_SCANORDER=”dm”

ORACLEASM_SCANEXCLUDE=”sd”

So, I have simply removed pair value to defualt,

ORACLEASM_SCANORDER=” ”

ORACLEASM_SCANEXCLUDE=” ”

3) restart the asmlib by (except 1st node):

# /etc/init.d/oracleasm restart

4) Scan ASM disks:

# /etc/init.d/oracleasm scandisks

Scanning the system for Oracle ASMLib disks:               [  OK  ]

8) List the ASM disks:

# /etc/init.d/oracleasm listdisks

DATA1

9) Now restart the CRS 

# crsctl stop crs -f

# crsctl start crs

# crsctl check cluster -all

**************************************************************

alpha:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

beta:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************