ORA-27300: OS System Dependent Operation Failed - Diagnose

ORA-27300: OS System Dependent Operation Failed

Error Overview

Error Text: ORA-27300: OS system dependent operation:<operation> failed with status: <N>

ORA-27300 is a wrapper error that Oracle raises when an OS-level system call returns an unexpected error code. It never appears alone in the alert log or trace file: it is always accompanied by a secondary Oracle error (usually ORA-27301 or ORA-27302) that names the failing operation, plus an OS error number (status: N) that maps to a Linux errno value.

Unlike ORA-27102 (memory), ORA-27123 (shmat), or ORA-27140 (semaphores), ORA-27300 can be caused by disk I/O failures, file permission problems, process resource exhaustion, or any other system call that Oracle wraps. The diagnostic approach is always the same: identify the operation name from the secondary error and decode the errno.

Understanding the Error Stack

Typical Error Stack in Alert Log

ORA-27300: OS system dependent operation:semop failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpwwait1

Each component:

• ORA-27300: Top-level wrapper — names the OS operation and the numeric errno
• ORA-27301: Human-readable OS error message (the strerror() text for the errno)
• ORA-27302: Internal Oracle location where the failure was detected (sskgp... functions relate to the Post/Wait IPC facility)

Common Operation Names in ORA-27300

Operation in error text	OS System Call	Likely Cause
semop	semop()	Semaphore limit exhausted — see ORA-27140
semget	semget()	Cannot create semaphore set — semmni limit
shmget	shmget()	Cannot create shared memory — see ORA-27125
shmat	shmat()	Cannot attach shared memory — see ORA-27123
mmap	mmap()	Virtual address space or /dev/shm full
open	open()	File permission or disk space problem
read / write	read() / write()	Disk I/O error or filesystem full
fork	fork()	Process limit reached (nproc)
pread / pwrite	pread64() / pwrite64()	Block device I/O error

Common Causes

1. errno 28 — No Space Left on Device (ENOSPC)

• The most common errno in ORA-27300 on production systems
• Can mean: disk filesystem full, /dev/shm full, semaphore namespace exhausted, or inode exhaustion
• Even if df -h shows free blocks, inodes may be at 100% (df -i)
• /dev/shm out of space when AMM is active

2. errno 12 — Cannot Allocate Memory (ENOMEM)

• The kernel refused a memory allocation (mmap, malloc, or shmget)
• Physical RAM plus swap is exhausted
• vm.overcommit_memory = 2 with overcommit_ratio set too low
• HugePages not available when use_large_pages = ONLY
• Relates to ORA-27102

3. errno 1 — Operation Not Permitted (EPERM)

• SELinux or AppArmor has denied the system call
• The oracle user does not have the required Linux capability
• Attempting mlock() without CAP_IPC_LOCK capability
• Kernel lockdown mode (secure boot with restricted kernel features)

4. errno 13 — Permission Denied (EACCES)

• IPC object permissions do not allow the oracle user to access them
• Shared memory segment or semaphore set created by a different user
• File system permission denying open/read/write to a datafile or control file
• Relates to ORA-27123

5. errno 11 — Resource Temporarily Unavailable (EAGAIN)

• A non-blocking operation found the resource busy
• Process table full (nproc ulimit or kernel pid_max exhausted)
• File lock already held by another process
• Temporary condition during high-load periods

6. errno 5 — Input/Output Error (EIO)

• Storage-layer error: disk failure, SAN fabric issue, or NFS timeout
• The kernel returned an I/O error from a read(), write(), or pread() call
• This is a hardware or network storage problem, not a configuration issue
• Requires immediate storage subsystem investigation

7. errno 22 — Invalid Argument (EINVAL)

• An invalid argument was passed to a system call
• Often seen when Oracle tries to attach to an orphaned or stale IPC object
• Semaphore set or shared memory segment key mismatch after a crash
• Wrong flags or size in a system call due to a version mismatch

Diagnostic Queries

Identify the Failing Operation and errno

# Step 1: Always start here — read the full error stack from the alert log
ALERT_LOG="$ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/alert_$ORACLE_SID.log"
grep -B2 -A10 "ORA-27300" $ALERT_LOG | tail -60

# Step 2: Decode the errno
#   errno 28 = ENOSPC  — No space left on device
#   errno 12 = ENOMEM  — Cannot allocate memory
#   errno  1 = EPERM   — Operation not permitted
#   errno 13 = EACCES  — Permission denied
#   errno 11 = EAGAIN  — Resource temporarily unavailable
#   errno  5 = EIO     — Input/output error
#   errno 22 = EINVAL  — Invalid argument
#   errno  2 = ENOENT  — No such file or directory
#   errno  4 = EINTR   — Interrupted system call

# Step 3: Look at the ORA-27302 location for context
#   sskgpwwait: wait for IPC event (semop)
#   sskgppost:  post IPC event (semop)
#   sskgpcreate: create IPC resources (shmget/semget)
#   ksepostevent: event posting (IPC layer)

Check Disk and Filesystem Space

# Filesystem space (blocks)
df -h

# Filesystem space (inodes) — often overlooked
df -i

# /dev/shm specifically (critical for AMM)
df -h /dev/shm
ls -la /dev/shm/

# Find large files consuming space
find /u01 /u02 /u03 -size +1G -type f 2>/dev/null | sort -k5 -rh | head -20

# Oracle trace and audit directories
du -sh $ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/
du -sh $ORACLE_BASE/admin/$ORACLE_SID/adump/ 2>/dev/null

Check Process and File Limits

# Oracle user ulimits
su - oracle -c "ulimit -a"

# Check system-wide process count
cat /proc/sys/kernel/pid_max
ps aux | wc -l

# File descriptor usage for Oracle processes
ls -la /proc/$(pgrep -f "ora_pmon_$ORACLE_SID" | head -1)/fd/ 2>/dev/null | wc -l
cat /proc/sys/fs/file-max
cat /proc/sys/fs/file-nr

# Open file limit for oracle processes
for pid in $(pgrep -u oracle); do
  fds=$(ls /proc/$pid/fd 2>/dev/null | wc -l)
  if [ $fds -gt 900 ]; then
    echo "PID $pid has $fds open file descriptors"
    cat /proc/$pid/cmdline | tr '\0' ' '
    echo ""
  fi
done

Check IPC Resources

# Full IPC status — semaphores, shared memory, message queues
ipcs -a

# Semaphore limits vs current usage
ipcs -ls
ipcs -s | grep oracle | wc -l

# Shared memory limits vs current usage
ipcs -lm
ipcs -m | grep oracle

# Message queue limits (rarely involved but worth checking)
ipcs -lq

Check OS Kernel Logs

# Recent kernel messages (last 100 lines)
dmesg | tail -100

# Filter for memory and IPC errors
dmesg | grep -iE "(oom|out of memory|killed|semaphore|shm|ipc|error|i/o error)"

# Systemd journal for kernel messages
journalctl -k --since "1 hour ago" | grep -iE "(error|fail|oom|shm|sem)"

# Check for OOM killer activity
dmesg | grep -i "oom_kill"
journalctl -k | grep "Out of memory: Kill"

# Storage/disk errors
dmesg | grep -iE "(i/o error|hard error|medium error|sata|scsi|nvme)" | tail -30

Oracle-Side Diagnostics

-- Check for recent alert log errors from Oracle's perspective
SELECT
  originating_timestamp,
  message_text
FROM v$diag_alert_ext
WHERE originating_timestamp > SYSTIMESTAMP - INTERVAL '1' HOUR
  AND (message_text LIKE '%ORA-27300%'
    OR message_text LIKE '%ORA-27301%'
    OR message_text LIKE '%ORA-27302%')
ORDER BY originating_timestamp DESC;

-- Check for process-level errors
SELECT
  p.pid,
  p.spid,
  p.program,
  p.background,
  p.error
FROM v$process p
WHERE p.error IS NOT NULL
ORDER BY p.pid;

-- Check wait events for IPC/OS-related waits
SELECT
  event,
  total_waits,
  total_timeouts,
  ROUND(time_waited/100, 2) AS time_waited_sec,
  ROUND(average_wait/100, 4) AS avg_wait_sec
FROM v$system_event
WHERE event LIKE '%IPC%'
   OR event LIKE '%os%'
   OR event LIKE '%post%'
ORDER BY total_waits DESC;

Step-by-Step Resolution

Step 1: Decode the Error Stack

# Collect the full error context
ALERT_LOG="$ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/alert_$ORACLE_SID.log"

echo "=== Last ORA-27300 occurrence ==="
grep -n "ORA-27300\|ORA-27301\|ORA-27302" $ALERT_LOG | tail -10

echo ""
echo "=== Full context around last occurrence ==="
LINE=$(grep -n "ORA-27300" $ALERT_LOG | tail -1 | cut -d: -f1)
sed -n "$((LINE-10)),$((LINE+15))p" $ALERT_LOG

Step 2: Route to the Specific Fix

If errno = 28 (ENOSPC) and operation = semop/semget:

# Semaphore namespace is full — clean orphans and increase semmns
ipcs -s | grep oracle | awk '{print $2}' | xargs -I{} ipcrm -s {}
sudo sysctl -w kernel.sem="250 32000 100 256"
echo "kernel.sem = 250 32000 100 256" | sudo tee -a /etc/sysctl.conf
# See also ORA-27140 guide for semaphore sizing

If errno = 28 (ENOSPC) and operation = mmap/shmget:

# /dev/shm is full
df -h /dev/shm
ls -lah /dev/shm/
sudo mount -o remount,size=20G /dev/shm
# See also ORA-27125 and ORA-00845 guides

If errno = 28 (ENOSPC) and operation = open/write:

# Filesystem is full
df -h
df -i
# Free space: purge Oracle trace files, audit logs, archived logs
find $ORACLE_BASE/diag -name "*.trc" -mtime +7 -delete
find $ORACLE_BASE/diag -name "*.trm" -mtime +7 -delete

If errno = 12 (ENOMEM):

# Memory allocation failed
free -h
# Increase kernel.shmmax / adjust Oracle SGA size
# See ORA-27102 guide for complete resolution
sudo sysctl -w kernel.shmmax=17179869184

If errno = 1 (EPERM) or 13 (EACCES):

# Permissions problem
getenforce         # SELinux?
id oracle          # Right groups?
stat /dev/shm      # /dev/shm permissions?
# See ORA-27123 guide for permissions resolution

If errno = 5 (EIO):

# Storage I/O error — hardware investigation required
dmesg | grep -iE "(i/o error|hard error|scsi|nvme|sata)" | tail -30
# Check SAN/NAS fabric, disk health
smartctl -a /dev/sda   # If local disk
# Contact storage team immediately

If errno = 11 (EAGAIN):

# Process or resource limit
ulimit -u            # max user processes
cat /proc/sys/kernel/pid_max
ps aux | wc -l       # current process count
# Increase nproc ulimit or pid_max
sudo sysctl -w kernel.pid_max=65536

Step 3: Collect Trace File Evidence

# Find the trace file generated at the time of the error
ls -lt $ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/*.trc \
  | head -10

# Read the most recent trace
LATEST_TRC=$(ls -t $ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/*.trc | head -1)
cat $LATEST_TRC | head -100

# Use Oracle's adrci for a formatted incident report
adrci << 'EOF'
SET HOME diag/rdbms/${ORACLE_SID}/${ORACLE_SID}
SHOW INCIDENT -MODE DETAIL -P "INCIDENT_ID > 0"
EXIT;
EOF

Step 4: Apply the Fix and Test

# After applying the appropriate fix (see Step 2), attempt Oracle startup
sqlplus / as sysdba << 'EOF'
STARTUP;
SELECT status FROM v$instance;
EXIT;
EOF

Prevention Strategies

ORA-27300 Prevention Monitoring Script

#!/bin/bash
# oracle_os_health.sh — Proactive OS health checks to prevent ORA-27300

ORACLE_SID=${1:-ORCL}
WARN=0

echo "=== Oracle OS Health Check: $(date) ==="

# 1. Filesystem space
while read fs size used avail pct mount; do
  PCT=${pct/\%/}
  if [ "$PCT" -ge 85 ] 2>/dev/null; then
    echo "[WARN] Filesystem $mount is ${pct} full ($avail available)"
    WARN=$((WARN+1))
  fi
done < <(df -h | tail -n +2)

# 2. Inode usage
while read fs inodes used avail pct mount; do
  PCT=${pct/\%/}
  if [ "$PCT" -ge 85 ] 2>/dev/null; then
    echo "[WARN] Inode exhaustion on $mount: ${pct} used"
    WARN=$((WARN+1))
  fi
done < <(df -i | tail -n +2)

# 3. /dev/shm
SHM_PCT=$(df /dev/shm | awk 'NR==2{print $5}' | tr -d '%')
if [ "$SHM_PCT" -ge 70 ]; then
  echo "[WARN] /dev/shm is ${SHM_PCT}% full"
  WARN=$((WARN+1))
fi

# 4. Semaphores
SEM_USED=$(ipcs -s | grep oracle | wc -l)
read SEMMSL SEMMNS SEMOPM SEMMNI < /proc/sys/kernel/sem
SEM_PCT=$(( SEM_USED * 100 / SEMMNI ))
if [ $SEM_PCT -ge 70 ]; then
  echo "[WARN] Semaphore sets: $SEM_USED / $SEMMNI (${SEM_PCT}% used)"
  WARN=$((WARN+1))
fi

# 5. Memory
MEM_FREE=$(grep MemAvailable /proc/meminfo | awk '{print $2}')
MEM_TOTAL=$(grep MemTotal /proc/meminfo | awk '{print $2}')
MEM_PCT=$(( (MEM_TOTAL - MEM_FREE) * 100 / MEM_TOTAL ))
if [ $MEM_PCT -ge 90 ]; then
  echo "[WARN] Memory ${MEM_PCT}% used (only ${MEM_FREE}KB available)"
  WARN=$((WARN+1))
fi

# 6. OOM killer activity
OOM_EVENTS=$(dmesg --since "10 minutes ago" 2>/dev/null | grep -c "Out of memory" || \
             dmesg | tail -200 | grep -c "Out of memory")
if [ "$OOM_EVENTS" -gt 0 ]; then
  echo "[WARN] OOM killer has fired $OOM_EVENTS time(s) recently"
  WARN=$((WARN+1))
fi

echo ""
if [ $WARN -eq 0 ]; then
  echo "[OK] All OS health checks passed"
else
  echo "[!!] $WARN warning(s) found — investigate before they cause ORA-27300"
fi

Alert Log Monitoring for ORA-27300

-- Schedule regular alert log scan for ORA-27300 occurrences
CREATE OR REPLACE PROCEDURE check_ora27300_in_alertlog AS
  v_count NUMBER;
BEGIN
  SELECT COUNT(*) INTO v_count
  FROM v$diag_alert_ext
  WHERE originating_timestamp > SYSTIMESTAMP - INTERVAL '1' HOUR
    AND message_text LIKE '%ORA-27300%';

  IF v_count > 0 THEN
    -- In production: raise an alert via DBMS_AQ or email
    DBMS_OUTPUT.PUT_LINE('ALERT: ' || v_count ||
      ' ORA-27300 occurrences in the last hour. Investigate OS resources immediately.');
  END IF;
END;
/

-- Schedule to run every 15 minutes
BEGIN
  DBMS_SCHEDULER.CREATE_JOB(
    job_name        => 'CHECK_ORA27300',
    job_type        => 'STORED_PROCEDURE',
    job_action      => 'check_ora27300_in_alertlog',
    repeat_interval => 'FREQ=MINUTELY;INTERVAL=15',
    enabled         => TRUE,
    comments        => 'Alert on ORA-27300 OS failures'
  );
END;
/

errno Reference Card

# Quick errno lookup — paste into runbook
cat << 'EOF'
ORA-27300 errno Quick Reference
================================
errno  1  EPERM   — Operation not permitted (SELinux, capabilities)
errno  2  ENOENT  — No such file or directory (missing file/device)
errno  4  EINTR   — Interrupted system call (signal during syscall)
errno  5  EIO     — I/O error (disk/SAN failure) — ESCALATE IMMEDIATELY
errno 11  EAGAIN  — Resource temporarily unavailable (process/fd limit)
errno 12  ENOMEM  — Cannot allocate memory (RAM exhausted)
errno 13  EACCES  — Permission denied (IPC or file permissions)
errno 22  EINVAL  — Invalid argument (stale/orphaned IPC object)
errno 28  ENOSPC  — No space left on device (disk, /dev/shm, or IPC namespace)
errno 30  EROFS   — Read-only file system (filesystem remounted read-only after I/O error)
errno 36  ENAMETOOLONG — File name too long
EOF

Related Errors

• ORA-27102 - Out of memory (ENOMEM from shmget/mmap — errno 12)
• ORA-27123 - Unable to attach to shared memory segment (EACCES/ENOMEM — errno 13/12)
• ORA-27125 - Unable to create shared memory segment (errno in shmget)
• ORA-27140 - Attach to post/wait facility failed (ENOSPC in semop — errno 28)
• ORA-00845 - MEMORY_TARGET not supported (/dev/shm ENOSPC)
• ORA-07445 - Exception encountered (kernel signal from OS fault)
• ORA-00600 - Internal error (can wrap OS errors at lower layers)

Emergency Response

Quick Triage Sequence

1. Read the error stack — do not skip this step

   tail -100 $ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/alert_$ORACLE_SID.log \
     | grep -A5 "ORA-27300"

2. Map errno to resource

   # The status number after "failed with status:" is the errno
   # See the errno reference card above

3. Check the most likely suspects in order

   df -h && df -i             # disk space and inodes (errno 28, 30)
   df -h /dev/shm             # /dev/shm for AMM (errno 28, 12)
   free -h                    # RAM (errno 12)
   ipcs -ls && ipcs -lm       # IPC limits (errno 28, 22)
   dmesg | tail -30           # kernel messages (errno 5 = EIO = URGENT)

4. Apply the targeted fix (see Step-by-Step Resolution above)

5. Restart Oracle and monitor

   STARTUP;
   SELECT status FROM v$instance;

If errno = 5 (EIO) — Escalation Required

# I/O error means storage hardware failure — this is a P1 incident
dmesg | grep -iE "(i/o error|hard error|scsi error|nvme error)" | tail -20
# Immediately:
# 1. Alert the storage/infrastructure team
# 2. Check for datafile corruption: SELECT * FROM v$recover_file;
# 3. Prepare for potential recovery from RMAN backup
# 4. Do NOT attempt further database startup until storage is confirmed healthy