10 09 2020

Recently encountered this on a server when I logged in, and then when trying to run commands

$ ssh 10.10.123.3
Last login: Tue Jul 14 18:55:19 2020 from  10.10.123.2
-bash: fork: Cannot allocate memory
# top 
-bash: fork: Cannot allocate memory
# clear
-bash: fork: Cannot allocate memory

To fix this, without rebooting the server I invoked the out of memory killer using sysrq.

First we need to enable it

echo 1 > /proc/sys/kernel/sysrq

Then invoke OOM killer

echo f > /proc/sysrq-trigger

This should then trigger some output as things get killed

[6044378.377748] Out of memory: Kill process 14435 (dsm_om_connsvcd) score 2 or sacrifice child
[6044378.377791] Killed process 14435 (dsm_om_connsvcd), UID 0, total-vm:6075028kB, anon-rss:256556kB, file-rss:9424kB, shmem-rss:0kB
[6044378.377827] systemd-journald[747]: /dev/kmsg buffer overrun, some messages lost.

Then disable sysrq

echo 0 > /proc/sys/kernel/sysrq

You should then be able to run commands, in this case I ran w and was confronted with massive load averages, which thankfully did settle back down.

# w
 18:53:39 up 69 days, 22:52,  1 user,  load average: 6039.00, 6038.95, 6038.89
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    10.10.123.2      18:48    3.00s  0.39s  0.37s w

Further reading – https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html

CRITICAL – hpasmd needs to be restarted, System: ‘unknown’, S/N: ‘unknown’, ROM: ‘unknown’ VCDS and Virtualbox