ORA-27140: Post/Wait Facility Failed - Fix Semaphore Limits

ORA-27140: Attach to Post/Wait Facility Failed

Error Overview

Error Text: ORA-27140: attach to post/wait facility failed

ORA-27140 is a startup-phase error that occurs when Oracle cannot attach to the Inter-Process Communication (IPC) semaphore facility used for the Post/Wait mechanism. Oracle’s background processes communicate using semaphores: one process “posts” (signals) another by incrementing a semaphore, and the receiver “waits” on that semaphore. If the OS cannot create or attach to the required semaphore set, no inter-process coordination is possible and startup fails immediately.

This error is almost always caused by Linux kernel semaphore limits being too small for the number of Oracle processes configured. The key parameter is PROCESSES in the SPFILE: each Oracle process (foreground and background) requires at least one semaphore operation slot.

Common Causes

1. kernel.sem semmsl Too Small

• semmsl is the maximum number of semaphores per semaphore set
• Oracle creates a semaphore set and needs at least PROCESSES + headroom semaphores in it
• Default values on older Linux distributions (e.g., 250) can be insufficient for large PROCESSES values
• When PROCESSES=1000 and semmsl=250, Oracle cannot create a sufficiently large semaphore set

2. kernel.sem semmns Exhausted

• semmns is the system-wide total number of semaphores across all semaphore sets
• Multiple Oracle instances on the same host each consume semaphores proportional to their PROCESSES parameter
• Non-Oracle software (PostgreSQL, application servers, message queues) also consumes semmns
• When the cumulative demand exceeds semmns, new semaphore creation fails

3. kernel.sem semmni Limit Reached

• semmni is the maximum number of semaphore sets (not individual semaphores)
• Each Oracle instance creates one or more semaphore sets
• On dense servers with many instances, semmni can be exhausted
• The default value of 128 on some distributions supports far fewer Oracle instances than expected

4. kernel.sem semopm Too Small

• semopm is the maximum number of semaphore operations per semop() system call
• Oracle performs batch semaphore operations during startup
• If semopm is smaller than the number of operations Oracle needs in a single call, startup fails
• Oracle recommends semopm >= 100

5. Orphaned Semaphore Sets from a Previous Crash

• After an abnormal instance termination, semaphore sets remain in the kernel
• These orphaned sets consume semmns and semmni quota
• On restart, Oracle may find the pool exhausted by its own orphaned resources
• Unlike shared memory, semaphores do not have a filesystem representation and require ipcrm to clean

6. PROCESSES Parameter Set Too High

• An overly large PROCESSES value drives the semaphore requirement beyond what the OS can provide
• This is common when a DBA copies a large-system SPFILE to a smaller server
• The fix may be to reduce PROCESSES rather than increase OS limits

Diagnostic Queries

Check Oracle PROCESSES Parameter

-- Current PROCESSES setting
SELECT value AS processes_limit
FROM v$parameter
WHERE name = 'processes';

-- How many processes are actually in use right now
SELECT COUNT(*) AS active_processes
FROM v$process;

-- Process headroom
SELECT
  p.value                          AS max_processes,
  COUNT(s.sid)                     AS current_sessions,
  p.value - COUNT(s.sid)           AS headroom
FROM v$parameter p
CROSS JOIN v$session s
WHERE p.name = 'processes'
GROUP BY p.value;

-- Background processes requiring semaphores
SELECT name, description, background, paddr
FROM v$bgprocess
WHERE paddr != '00'
ORDER BY name;

Check Semaphore Usage (OS Commands)

# The four kernel.sem values: semmsl semmns semopm semmni
cat /proc/sys/kernel/sem

# Formatted display
IFS=' ' read -r semmsl semmns semopm semmni < /proc/sys/kernel/sem
echo "semmsl (max semaphores per set): $semmsl"
echo "semmns (total semaphores system): $semmns"
echo "semopm (max ops per semop call): $semopm"
echo "semmni (max semaphore sets):      $semmni"

# Current semaphore usage summary
ipcs -ls

# All semaphore sets with owner, permissions, and count
ipcs -s

# Detailed view of each semaphore set
ipcs -s -i <semid>   # Replace <semid> with actual ID

# Count currently allocated semaphore sets
ipcs -s | grep -c oracle

# Count Oracle-owned sets vs total
echo "Oracle semaphore sets: $(ipcs -s | grep oracle | wc -l)"
echo "Total semaphore sets:  $(ipcs -s | grep -v "^--\|Semaphore\|key" | wc -l)"

Calculate Required Semaphore Values

# Rule of thumb from Oracle documentation:
# semmsl >= PROCESSES + 10
# semmns >= (semmsl * semmni) — but practical: num_instances * (PROCESSES + 10) * 2
# semopm >= 100
# semmni >= num_oracle_instances * 2 + headroom

# Read current PROCESSES from SPFILE when DB is down
PROCESSES=$(strings $ORACLE_HOME/dbs/spfile${ORACLE_SID}.ora 2>/dev/null \
  | grep -i "^.*\.processes=" | awk -F'=' '{print $2}' | tr -d '[:space:]')
echo "PROCESSES parameter: ${PROCESSES:-unknown}"

# Calculate minimum semmsl
MIN_SEMMSL=$(( ${PROCESSES:-500} + 10 ))
echo "Minimum semmsl: $MIN_SEMMSL"

# Calculate recommended semmns for this instance
INSTANCES=1  # Adjust for RAC or multiple instances
MIN_SEMMNS=$(( INSTANCES * (${PROCESSES:-500} + 10) * 2 ))
echo "Minimum semmns (for $INSTANCES instance(s)): $MIN_SEMMNS"

Examine Alert Log for Semaphore Errors

ALERT_LOG="$ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/alert_$ORACLE_SID.log"

# Find ORA-27140 with surrounding context
grep -B5 -A10 "ORA-27140" $ALERT_LOG | tail -60

# Look for the secondary OS error code
grep -A3 "ORA-27140" $ALERT_LOG | tail -30

# Common secondary errors:
# Linux-X86_64 Error: 28: No space left on device -> semmns or semmni exhausted
# Linux-X86_64 Error: 22: Invalid argument         -> semmsl too small or bad semaphore ID
# Linux-X86_64 Error: 13: Permission denied        -> IPC permissions problem
# Linux-X86_64 Error:  1: Operation not permitted  -> SELinux or capability restriction

# Check for orphaned semaphore messages
dmesg | grep -i semaphore
journalctl -k --since "1 hour ago" | grep -iE "(sem|ipc)"

Step-by-Step Resolution

Step 1: Read the Current kernel.sem Values

cat /proc/sys/kernel/sem
# Output: semmsl  semmns  semopm  semmni
# Example:   250   32000     100     128
# This means: 250 sems/set, 32000 total, 100 ops/call, 128 sets max

Step 2: Calculate Required Values

# Get PROCESSES from a running instance (if available)
sqlplus -S / as sysdba << 'EOF'
SET HEADING OFF FEEDBACK OFF PAGESIZE 0
SELECT value FROM v$parameter WHERE name = 'processes';
EXIT;
EOF

# Or read from SPFILE directly
PROCESSES=1000   # Use your actual value

# Oracle's recommended formula:
SEMMSL=$(( PROCESSES + 10 ))        # e.g., 1010
SEMMNI=128                           # increase if multiple instances
SEMMNS=$(( SEMMSL * SEMMNI ))       # e.g., 128 * 1010 = 129280
SEMOPM=100

echo "Recommended kernel.sem: $SEMMSL $SEMMNS $SEMOPM $SEMMNI"

Step 3: Apply the New Semaphore Parameters

# Apply immediately (effective until next reboot)
sudo sysctl -w kernel.sem="$SEMMSL $SEMMNS $SEMOPM $SEMMNI"

# Verify
cat /proc/sys/kernel/sem

# Common production-grade values that satisfy most Oracle deployments:
sudo sysctl -w kernel.sem="250 32000 100 128"
# For PROCESSES > 500 or multiple instances:
sudo sysctl -w kernel.sem="500 32000 100 256"

Step 4: Persist the Changes

# Back up sysctl.conf
sudo cp /etc/sysctl.conf /etc/sysctl.conf.bak.$(date +%Y%m%d)

# Remove any old kernel.sem line and add the new one
sudo sed -i '/^kernel\.sem/d' /etc/sysctl.conf
echo "kernel.sem = 250 32000 100 256" | sudo tee -a /etc/sysctl.conf

# Apply and verify
sudo sysctl -p
cat /proc/sys/kernel/sem

Step 5: Remove Orphaned Semaphore Sets

# List oracle-owned semaphore sets
ipcs -s | grep oracle

# Remove each orphaned set (those with 0 attached processes)
ipcs -s | grep oracle | awk '{print $2}' | while read semid; do
  echo "Removing semaphore set $semid"
  ipcrm -s "$semid" && echo "  OK" || echo "  Failed (try with sudo)"
done

# Verify cleanup
ipcs -s | grep oracle

Step 6: Restart Oracle

-- Attempt startup
STARTUP;

-- Verify background processes are all running
SELECT name, description, error
FROM v$bgprocess
WHERE paddr != '00'
ORDER BY name;

-- Confirm PROCESSES headroom is comfortable
SELECT
  p.value AS max_processes,
  COUNT(s.paddr) AS current_processes
FROM v$parameter p, v$process s
WHERE p.name = 'processes'
GROUP BY p.value;

Prevention Strategies

Recommended kernel.sem for Oracle Deployments

# Oracle Installation Guide recommended values (OL7/RHEL7 and later)
# For a single instance with PROCESSES up to 500:
kernel.sem = 250 32000 100 128

# For PROCESSES 500–1500 or 2–4 instances:
kernel.sem = 500 32000 100 256

# For PROCESSES > 1500 or RAC with many nodes:
kernel.sem = 1000 32000 100 512

# Calculate precisely: semmsl = PROCESSES + 10; semmni = instances * 20
# Then: semmns = semmsl * semmni

Semaphore Capacity Planning Script

#!/bin/bash
# oracle_sem_capacity.sh — Check semaphore capacity for planned workload

INSTANCES=${1:-1}
PROCESSES_PER=${2:-500}

# Current kernel limits
read SEMMSL SEMMNS SEMOPM SEMMNI < /proc/sys/kernel/sem
echo "=== Current Kernel Semaphore Limits ==="
echo "semmsl (max per set):  $SEMMSL"
echo "semmns (system total): $SEMMNS"
echo "semopm (max ops/call): $SEMOPM"
echo "semmni (max sets):     $SEMMNI"
echo ""

# Requirements
REQ_SEMMSL=$(( PROCESSES_PER + 10 ))
REQ_SEMMNI=$(( INSTANCES * 2 + 10 ))
REQ_SEMMNS=$(( REQ_SEMMSL * REQ_SEMMNI ))
echo "=== Requirements for $INSTANCES instance(s) with PROCESSES=$PROCESSES_PER ==="
echo "Required semmsl: $REQ_SEMMSL  (current: $SEMMSL)"
echo "Required semmni: $REQ_SEMMNI  (current: $SEMMNI)"
echo "Required semmns: $REQ_SEMMNS  (current: $SEMMNS)"
echo ""

# Current usage
CURRENT_SETS=$(ipcs -s | grep oracle | wc -l)
echo "=== Current Semaphore Usage ==="
echo "Oracle semaphore sets in use: $CURRENT_SETS / $SEMMNI"
echo ""

# Verdict
FAIL=0
[ $SEMMSL -ge $REQ_SEMMSL ] || { echo "[FAIL] semmsl too small"; FAIL=1; }
[ $SEMMNI -ge $REQ_SEMMNI ] || { echo "[FAIL] semmni too small"; FAIL=1; }
[ $SEMMNS -ge $REQ_SEMMNS ] || { echo "[FAIL] semmns too small"; FAIL=1; }
[ $SEMOPM -ge 100 ]          || { echo "[FAIL] semopm < 100"; FAIL=1; }
[ $FAIL -eq 0 ] && echo "[OK] Semaphore limits are sufficient" \
  || echo ""
echo "Suggested fix: sudo sysctl -w kernel.sem=\"$REQ_SEMMSL $REQ_SEMMNS 100 $REQ_SEMMNI\""

Monitor Semaphore Consumption

-- Track process count trends to predict when limits will be approached
SELECT
  TO_CHAR(sample_time, 'YYYY-MM-DD HH24') AS sample_hour,
  MAX(session_count) AS peak_sessions
FROM (
  SELECT
    s.sample_time,
    COUNT(*) AS session_count
  FROM dba_hist_active_sess_history s
  WHERE s.sample_time > SYSDATE - 7
  GROUP BY s.sample_time
)
GROUP BY TO_CHAR(sample_time, 'YYYY-MM-DD HH24')
ORDER BY sample_hour DESC
FETCH FIRST 24 ROWS ONLY;

-- Alert if approaching PROCESSES limit
SELECT
  p.value            AS processes_limit,
  COUNT(pr.pid)      AS processes_active,
  ROUND(COUNT(pr.pid) / p.value * 100, 1) AS pct_used,
  CASE WHEN COUNT(pr.pid) / p.value > 0.80 THEN 'WARNING'
       ELSE 'OK' END AS status
FROM v$parameter p
JOIN v$process pr ON 1=1
WHERE p.name = 'processes'
GROUP BY p.value;

Automated Post-Crash Semaphore Cleanup

#!/bin/bash
# oracle_sem_cleanup.sh — Called from dbshut or crash handler

ORACLE_USER=${1:-oracle}

echo "$(date): Starting semaphore cleanup for user $ORACLE_USER"

# Remove semaphore sets owned by oracle with no attached processes
REMOVED=0
for semid in $(ipcs -s | grep "$ORACLE_USER" | awk '{print $2}'); do
  # semaphore sets don't have nattch, so clean all owned by oracle when DB is down
  if ! pgrep -u "$ORACLE_USER" -f "ora_smon" > /dev/null 2>&1; then
    ipcrm -s "$semid" 2>/dev/null && REMOVED=$((REMOVED+1))
  fi
done

echo "$(date): Removed $REMOVED orphaned semaphore sets"
ipcs -s | grep "$ORACLE_USER" | wc -l

Related Errors

• ORA-27102 - Out of memory (kernel refused memory allocation)
• ORA-27123 - Unable to attach to shared memory segment
• ORA-27125 - Unable to create shared memory segment
• ORA-27300 - OS system dependent operation failed (generic OS wrapper)
• ORA-00845 - MEMORY_TARGET not supported on this system
• ORA-00020 - Maximum number of processes exceeded (related PROCESSES limit)

Emergency Response

Quick Fixes

1. Read the current semaphore state

   cat /proc/sys/kernel/sem
   ipcs -ls
   ipcs -s | grep oracle | wc -l

2. Apply minimum safe values immediately

   sudo sysctl -w kernel.sem="250 32000 100 256"

3. Remove orphaned Oracle semaphore sets

   ipcs -s | grep oracle | awk '{print $2}' | xargs -I{} ipcrm -s {}

4. If PROCESSES is too high for the server, reduce it

   strings $ORACLE_HOME/dbs/spfile${ORACLE_SID}.ora | grep processes
   sqlplus / as sysdba
   ALTER SYSTEM SET processes = 300 SCOPE = SPFILE;
   STARTUP;

5. Persist the fix before the next reboot

   echo "kernel.sem = 250 32000 100 256" | sudo tee -a /etc/sysctl.conf
   sudo sysctl -p

Post-Resolution Verification

# Confirm live kernel values
cat /proc/sys/kernel/sem

# Confirm Oracle is using semaphores
ipcs -s | grep oracle

# Confirm all Oracle background processes are up
sqlplus -S / as sysdba << 'EOF'
SELECT COUNT(*) || ' background processes running' AS status
FROM v$bgprocess WHERE paddr != '00';
EXIT;
EOF