How to Explain RAC Node Eviction in a Client Interview – A Complete Guide for DBAs

If you are preparing for a senior Oracle DBA interview, one question that comes up almost every time is: "What is RAC node eviction and how do you troubleshoot it?" This is not a question you can answer with a one-liner. Interviewers expect you to demonstrate deep understanding of Oracle Clusterware internals, network architecture, and real-world troubleshooting methodology. In this comprehensive guide, we break down exactly how to structure your answer so that you sound confident, technically precise, and interview-ready.

What Is RAC Node Eviction? Start With the Definition

In an Oracle Real Application Clusters (RAC) environment, node eviction is the process by which the cluster forcibly removes a node that is no longer responding or is behaving abnormally. This is a protective mechanism — not a failure — designed to maintain data integrity and prevent a catastrophic scenario known as split-brain.

When answering in an interview, start by framing node eviction as a cluster health mechanism rather than an error. Say something like: "Node eviction is the Oracle Clusterware's self-healing response when it detects that a member node can no longer participate reliably in the cluster. The cluster evicts the problematic node to protect the remaining nodes and ensure data consistency across shared storage."

Why Does Node Eviction Happen? The Key Causes

Interviewers love follow-up questions about causes. Here are the primary reasons a node gets evicted from a RAC cluster, and how to explain each one clearly.

1. Private Interconnect Failure

The private interconnect is the dedicated high-speed network used for inter-node communication including cache fusion block transfers and cluster heartbeat messages. When this link fails or becomes unreliable, nodes can no longer communicate with each other. The Cluster Synchronization Services daemon (CSSD) misses heartbeat messages, and if the misscount threshold is exceeded, the cluster triggers a node eviction. In your interview, explain that the default misscount is typically 30 seconds, and the CSSD checks for heartbeats every second. If heartbeats are missed for misscount consecutive seconds, the node is considered dead.

2. Voting Disk Inaccessibility

Voting disks are the arbitration mechanism that Oracle Clusterware uses to determine which nodes are alive and which should be evicted. Each node periodically writes a heartbeat to the voting disk. If a node cannot access the voting disk within the disktimeout period (default 200 seconds), or if it loses the majority of voting disks, it will be evicted. In your answer, mention that Oracle recommends an odd number of voting disks (typically 3 or 5) so that majority-based decisions can be made cleanly. With Oracle 11gR2 and later, voting disks are typically stored in ASM disk groups managed by Oracle ASM.

3. Operating System or Hardware Hang

If the operating system on a node hangs due to a kernel panic, excessive CPU usage, memory exhaustion, or a runaway process, the CSSD process may fail to send heartbeats. From the surviving nodes' perspective, the hung node is indistinguishable from a dead node. The cluster will evict it. A strong interview answer would include: "I have seen scenarios where a poorly configured HugePages setup or a memory leak in a third-party application caused the OS to become unresponsive, which in turn caused the CSSD to miss heartbeats and triggered an eviction."

4. CSSD Process Failure or Timeout

The Cluster Synchronization Services Daemon (ocssd.bin) runs as a real-time priority process on each RAC node. If this process is killed, crashes, or is unable to run due to scheduling delays, the node will reboot itself. This is by design — Oracle intentionally makes CSSD termination fatal to the node because if CSSD is not running, the node cannot participate in cluster membership. Explain in the interview that this self-eviction is a safety net: it is better for a node to reboot itself than to continue operating without cluster coordination, which could lead to data corruption.

5. Split-Brain Resolution

Split-brain is the most dangerous scenario in any clustered environment. It occurs when the interconnect fails and each partition of nodes believes it is the surviving cluster. Both partitions could attempt to write to shared storage simultaneously, leading to data corruption. Oracle Clusterware resolves split-brain by using voting disks as the tiebreaker. The partition that can access the majority of voting disks survives, and the minority partition's nodes are evicted. In a two-node cluster, this becomes especially critical because there is no natural majority. In this case, the node with the lowest node number or the one that can reach the voting disk first typically survives.

The Node Eviction Process: Step by Step

When explaining the eviction process in an interview, walking through the sequence of events shows that you truly understand the internals. Here is the step-by-step flow:

Step 1 — Heartbeat failure detected: The CSSD on surviving nodes detects that it has not received a network heartbeat from the suspect node within the misscount window. Step 2 — Disk heartbeat check: Before evicting, the surviving nodes check whether the suspect node is still writing heartbeats to the voting disk. If the disk heartbeats are also missing, the node is confirmed as dead. Step 3 — Cluster reconfiguration initiated: The surviving nodes begin a cluster membership reconfiguration. They vote on which nodes should remain in the cluster using the voting disk. Step 4 — IPMI/BMC-based node kill (if configured): Starting from Oracle 11gR2, if IPMI or BMC is configured, the surviving nodes can issue a hardware-level reset to the evicted node to ensure it is truly stopped. Step 5 — CSS reconfig complete: The cluster completes reconfiguration with the reduced membership. Instance recovery is triggered by the surviving instances to roll back uncommitted transactions. Step 6 — Services relocated: VIPs and database services that were running on the evicted node are failed over to the surviving nodes based on the configured service placement policy.

Key Parameters Every DBA Should Know

In interviews, being able to cite specific parameter names and their default values demonstrates hands-on experience. Here are the critical parameters related to node eviction:

misscount — Default 30 seconds. The maximum time CSS waits for a network heartbeat before declaring a node dead. disktimeout — Default 200 seconds. The maximum time a node can go without successfully completing a voting disk I/O before it evicts itself. reboottime — Default 3 seconds. The time CSS waits after deciding to evict a node before actually performing the reboot. You can check the current values using: crsctl get css misscount, crsctl get css disktimeout, and crsctl get css reboottime. Mentioning these commands in an interview shows practical experience.

How to Troubleshoot a Node Eviction — The Diagnostic Approach

This is where many candidates either shine or stumble. Interviewers want to know your systematic approach to diagnosing a node eviction after it has occurred.

Step 1: Check the CRS Alert Log

The first place to look is the CRS alert log located at $GRID_HOME/log/<hostname>/alertlog/alert.log. This log records the high-level events including when the eviction was detected and initiated. Look for messages containing keywords like evicted, reboot, node kill, or split brain.

Step 2: Examine the CSSD Trace Logs

The CSSD trace logs are the most critical logs for diagnosing eviction root cause. They are located at $GRID_HOME/log/<hostname>/cssd/ocssd.trc. These logs contain detailed timestamps showing when heartbeats were missed, which nodes detected the failure, and which voting disk I/Os succeeded or failed. Look for entries with NETWORK HB and DISK HB to trace the heartbeat timeline.

Step 3: Review OS System Logs

Check /var/log/messages or /var/log/syslog around the eviction timestamp. Look for hardware errors, NIC failures, storage timeout messages, kernel panics, or out-of-memory killer events. On many occasions, the root cause of a node eviction is found in the OS logs rather than the Oracle logs.

Step 4: Check Network and Interconnect Health

Run oifcfg getif to verify the current interconnect configuration. Check for packet loss and latency using ping and traceroute on the private interconnect. Review switch logs if available. On bonded interfaces, check if one of the underlying NICs failed causing a failover delay.

Step 5: Analyze Voting Disk I/O Performance

Use crsctl query css votedisk to list voting disks and their current status. Check storage-level performance metrics for the ASM disk group hosting the voting disks. If the storage array experienced latency spikes, the voting disk I/O could have timed out.

Sample Interview Answer: How to Put It All Together

Here is a model answer that you can adapt for your interviews. Practice delivering this in under 3 minutes:

Node eviction in Oracle RAC is a cluster self-protection mechanism. It occurs when Oracle Clusterware determines that a member node can no longer participate reliably in the cluster. The most common causes are private interconnect failures, voting disk I/O timeouts, OS-level hangs, and CSSD process failures. The eviction process involves the Cluster Synchronization Services daemon detecting missed heartbeats and then initiating a cluster reconfiguration that excludes the problematic node. The key parameters governing this behavior are misscount (default 30 seconds) and disktimeout (default 200 seconds). When I troubleshoot a node eviction, my approach is systematic: I start with the CRS alert log, then the CSSD trace logs, check OS system logs, verify interconnect health, and review storage I/O performance. In my experience, the root cause often turns out to be a network switch issue or storage latency spike rather than an Oracle-level problem.

Common Follow-Up Interview Questions on Node Eviction

Q: What happens to sessions connected to the evicted node? A: Active sessions are terminated. If Transparent Application Failover (TAF) or Application Continuity is configured, sessions can be automatically reconnected to a surviving instance. Otherwise, applications receive ORA-03113 or ORA-03135 errors and must reconnect manually.

Q: Can both nodes evict each other simultaneously? A: No. The voting disk arbitration mechanism prevents this. In a split-brain scenario, only the partition that can access the majority of voting disks survives. This is precisely why voting disks exist.

Q: How do you prevent node evictions proactively? A: Implement redundant private interconnect with NIC bonding or Oracle HAIP. Use multiple voting disks on separate storage controllers. Monitor interconnect latency and packet loss. Ensure proper OS tuning for RAC including HugePages, kernel parameters, and time synchronization. Keep Clusterware patched.

Q: What is the difference between misscount and disktimeout? A: misscount controls how long CSS waits for network heartbeats over the private interconnect. disktimeout controls how long CSS waits for successful I/O to the voting disk. A node can be evicted due to either threshold being exceeded.

Q: What role does IPMI play in node eviction? A: IPMI provides an out-of-band management channel that allows surviving nodes to physically reset an unresponsive node at the hardware level, ensuring the evicted node cannot continue writing to shared storage.

Real-World Scenario: Narrate Your Experience Using STAR Method

Interviewers value real-world experience. Structure your narration using the STAR method. Example: Situation — A 4-node RAC cluster running Oracle 19c on Exadata had intermittent evictions every 2-3 weeks during peak hours. Task — Identify the root cause and prevent future evictions. Action — Collected CSSD logs, OS sar data, and network statistics. CSSD logs showed missed network heartbeats but healthy disk heartbeats, pointing to an interconnect issue. Switch logs revealed CRC errors on the evicted node's private NIC port due to a faulty cable. Result — Replaced the cable and implemented redundant interconnects using Oracle HAIP. Zero unplanned evictions in the following 6 months.

Interview Tips: Dos and Don'ts

Do: Use specific parameter names like misscount and disktimeout. Mention log file locations and what to look for. Frame eviction as a protective mechanism. Describe your troubleshooting as systematic and log-driven. Reference real experience. Mention HAIP, NIC bonding, and redundant voting disks as prevention.

Don't: Give a one-line answer. Confuse misscount with disktimeout. Skip the split-brain explanation. Suggest increasing misscount to prevent evictions — this merely delays detection and increases corruption risk. Forget OS-level root causes.

Conclusion

RAC node eviction is one of the most important topics for any Oracle DBA interview because it tests your understanding of Clusterware architecture, network fundamentals, storage I/O, and systematic troubleshooting. By structuring your answer around the definition, causes, internal process, key parameters, and diagnostic methodology, you demonstrate both theoretical depth and practical expertise. Practice the model answer until you can deliver it naturally, and always be ready with a real-world scenario.

Want to master Oracle RAC and other advanced DBA topics for your next interview? Explore our comprehensive Oracle DBA training programs at ORADBA Online Training and Support. With over 12 years of training experience and students across the globe, we help DBAs crack interviews and advance their careers. Visit oracledbaonlinetraining.com to get started.