10 09 2020
-bash: fork: Cannot allocate memory
Recently encountered this on a server when I logged in, and then when trying to run commands
$ ssh 10.10.123.3 Last login: Tue Jul 14 18:55:19 2020 from 10.10.123.2 -bash: fork: Cannot allocate memory # top -bash: fork: Cannot allocate memory # clear -bash: fork: Cannot allocate memory
To fix this, without rebooting the server I invoked the out of memory killer using sysrq.
First we need to enable it
echo 1 > /proc/sys/kernel/sysrq
Then invoke OOM killer
echo f > /proc/sysrq-trigger
This should then trigger some output as things get killed
[6044378.377748] Out of memory: Kill process 14435 (dsm_om_connsvcd) score 2 or sacrifice child [6044378.377791] Killed process 14435 (dsm_om_connsvcd), UID 0, total-vm:6075028kB, anon-rss:256556kB, file-rss:9424kB, shmem-rss:0kB [6044378.377827] systemd-journald: /dev/kmsg buffer overrun, some messages lost.
Then disable sysrq
echo 0 > /proc/sys/kernel/sysrq
You should then be able to run commands, in this case I ran w and was confronted with massive load averages, which thankfully did settle back down.
# w 18:53:39 up 69 days, 22:52, 1 user, load average: 6039.00, 6038.95, 6038.89 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root pts/0 10.10.123.2 18:48 3.00s 0.39s 0.37s w
Further reading – https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html