How to Diagnose a Server That Keeps Crashing
%203.png&w=256&q=75){kind=small}
Build A Timeline Before You Touch Anything
A server that crashes once is a fluke. A server that crashes three times in a week is a pattern, and patterns have root causes you can find — but only if you know exactly when each crash happened. Before you check a single log file, pin down the timestamps.
Start with uptime to see how long the box has been up since its last boot, then pull the full reboot history with last -x reboot. Cross-reference those timestamps against journalctl --since and --until, or grep the raw timestamps out of /var/log/syslog if the system predates systemd's persistent journal. The goal is a simple list: date, time, what was running, what stopped responding first.
Check dmesg For OOM Kills First
The single most common cause of a server that keeps crashing is the Linux OOM (out-of-memory) killer terminating a critical process, over and over, every time available RAM hits zero. It's rarely logged anywhere obvious except dmesg, which is exactly why it gets missed.
Run dmesg -T | grep -i -E "killed process|out of memory" and match the timestamps against your crash timeline. If you see the same process — MySQL, PHP-FPM, Redis — getting killed repeatedly, that's your root cause, not a symptom. Once the kernel kills a database process mid-transaction, every service that depends on it fails a moment later, which is why a single OOM kill often looks like a full server outage.
Look For A Runaway Process With A Memory Leak
An OOM kill tells you memory ran out; it doesn't tell you why. Often the underlying cause is a single process with a memory leak — a stuck PHP-FPM worker, a WordPress plugin caching unboundedly, a Node process never releasing objects — that grows a little more after every request until it eventually eats all available RAM and takes the whole server down with it.
Run ps aux --sort=-%mem | head -15 right after a fresh boot, then again a few hours later, and compare the RSS column for the same process. A process whose memory climbs steadily and never drops back down between requests is the leak. On CloudStick, you don't need to script this comparison manually — the Server Stats section in the dashboard graphs RAM usage over time from the Zabbix agent, so a slow, sawtooth-free climb toward 100% is visible at a glance without SSHing in every hour to run ps.
Rule Out Disk Exhaustion
A full disk causes crashes that look identical to memory exhaustion from the outside — services stop responding, MySQL refuses writes, PHP-FPM can't write session files — but the fix is completely different, so check this before you assume it's RAM.
Run df -h now, but also check whether disk usage was at 100% specifically around your crash timestamps — a log file that grew unchecked, a backup that never cleaned up its previous archive, or a runaway error log from a misbehaving plugin can fill a volume in hours and then free up space again after rotation, making the cause invisible by the time you look.
Kernel Panics And Hardware Or Driver Faults
If the previous checks come up clean and you're on bare metal rather than a VPS, the next suspect is a kernel panic triggered by failing hardware or a misbehaving driver — this is less common on cloud VPS instances, where the underlying hardware is abstracted away, but it's a real cause on dedicated servers.
Search dmesg -T | grep -i panic and also check the IPMI or hardware vendor's out-of-band logs if available — panics often show a stack trace pointing at a specific driver or subsystem. Failing RAM is a frequent culprit and can be ruled in or out with a memtest86+ pass during a maintenance window. If your crashes correlate with load spikes rather than a fixed schedule, suspect a driver or firmware bug over hardware failure.
Don't Just Reboot And Hope
The instinctive move when a server crashes is to reboot it and move on the moment it comes back up. Resist that instinct until you've captured the evidence — rebooting before checking dmesg and /var/log/syslog destroys the exact data you need to find the root cause.
The kernel's dmesg ring buffer is limited in size and gets overwritten as new messages arrive, and a hard reboot can also trigger log rotation before you've had a chance to read the pre-crash entries. If a server just crashed and is still down or you can reach it in a degraded state, run dmesg -T and copy the last 200 lines to a file before you power-cycle anything. Once you reboot without checking, the exact evidence pointing to OOM kills, a kernel panic, or the specific process that failed first is often gone for good — and the next crash starts you from zero again.
The Systematic Checklist, Then Set Up Monitoring
Run through these four checks in order every time, before forming a theory: dmesg -T for OOM kills and panics, df -h for disk exhaustion at the crash timestamp, /var/log/syslog for the first service that failed (others usually cascade from that one root failure), and a memory-over-time comparison to catch a slow leak. Whichever check turns up an anomaly first is almost always your root cause — resist the urge to fix the second or third symptom you find instead of the first one.
Once you've found and fixed the immediate cause, the real fix is making sure the next crash arrives with data instead of guesswork. Set a memory and disk alert threshold so you're notified before a slow leak becomes an OOM kill, not after. CloudStick's Server Stats panel already tracks CPU, RAM, and disk usage continuously from the Zabbix agent — check it against your crash timeline the moment something goes down, and keep an eye on the trend line over the following days so a recurring leak shows up as a pattern, not a mystery, the second time around.
