Clusterware startup sequence for Oracle 11g R2

Understanding how the clusterware startup occurs is critical to the diagnosis and resolution of Oracle RAC problems.

In Unix and Linux operating systems, there is a master daemon process named INIT that functions to start up additional system background processes. The INIT process first spawns the init.ohasd process, which in turn starts up the Oracle High Availability Services Daemon (OHASD). In turn, the OHASD daemon then spawns additional Clusterware processes at each startup level as shown next:

• Level 1—OHASD spawns:

  • Cssdagent: Agent responsible for spawning CSSD

  • Orarootagent: Agent responsible for managing all root-owned ohasd resources

  • Oraagent: Agent responsible for managing all Oracle-owned ohasd resources

  • cssdmonitor: Monitors CSSD and node health (along wth the cssdagent)

• Level 2—OHASD rootagent spawns:

  • Cluster Ready Services Daemon (CRSD)—primary daemon responsible for managing cluster resources

  • Cluster Time Synchronization Services Daemon (CTSSD)

  • Diskmon—provides disk monitoring services

  • ASM Cluster File System (ACFS) Drivers 

During the second level of startup for Clusterware, the oraagent spawns the following Clusterware processes for 11g R2:

• MDNSD: Used for DNS lookup

• GIPCD: Used for inter-process and inter-node communication

• GPNPD: Grid Plug and Play Profile Daemon

• EVMD: Event Monitor Daemon

• ASM: Resource for monitoring ASM instances

• Level 3—CRSD spawns:

  • orarootagent: Agent responsible for managing all root-owned CRSD resources

  • oraagent: Agent responsible for managing all Oracle-owned CRSD resources

• Level 4—CRSD rootagent spawns:

  • Network resource: To monitor the public network

  • SCAN VIP(s): Single Client Access Name Virtual IPs

  • Node VIPs: One per node

  • ACFS Registery: For mounting ASM Cluster File system

  • GNS VIP (optional): VIP for GNS

During this phase for Clusterware startup with 11g R2, the oraagent spawns the following processes:

• ASM Resouce: ASM Instance(s) resource

• Diskgroup: Used for managing/monitoring ASM diskgroups 

• DB Resource: Used for monitoring and managing the DB and instances

• SCAN Listener: Listener for single client access name, listening on SCAN VIP

• Listener: Node listener listening on the Node VIP

• Services: Used for monitoring and managing services

• ONS: Oracle Notification Service

• eONS: Enhanced Oracle Notification Service

• GSD: For 9i backward compatibility

• GNS (optional): It is a grid naming service that performs name resolution

Log file locations for Oracle 11g RAC and ASM

The important Clusterware daemon logs are located under the <GRID_HOME>/log/<nodename> directory. There are additional logfiles located under the <GRID_HOME>/log/<nodename>directory as listed next:

alert<NODENAME>.log - look here first for most clusterware issues
./admin:
./agent:
./agent/crsd:
./agent/crsd/oraagent_oracle:
./agent/crsd/ora_oc4j_type_oracle:
./agent/crsd/orarootagent_root:
./agent/ohasd:
./agent/ohasd/oraagent_oracle:
./agent/ohasd/oracssdagent_root:
./agent/ohasd/oracssdmonitor_root:
./agent/ohasd/orarootagent_root:
./client:
./crsd:
./cssd:
./ctssd:
./diskmon:
./evmd:
./gipcd:
./gnsd:
./gpnpd:
./mdnsd:
./ohasd:
./racg:
./racg/racgeut:
./racg/racgevtf:
./racg/racgmain:
./srvm:

The cleanup of these logfiles can be automated with a shell script that can be scheduled to run from the cron facility. The cfgtoollogs directory located under the $GRID_HOME and $ORACLE_BASE directories contains other important logfiles such as the rootcrs.pl script and configuration assistants, such as the ASM Configuration Assistant (ASMCA).

ASM logs are located under $ORACLE_BASE/diag/asm/+asm/<ASM Instance Name>/trace

Oracle provides a useful script named diagcollection.pl, which can be found under the <GRID_HOME>/bin directory. This script, when executed, will automatically collect important Oracle 11g Clusterware files that can be sent to Oracle support engineers for analysis. You must be logged into Linux as the root user in order to run this script.