Linux Incident Response: Isolate a Compromised Server Without Downtime

A step-by-step Linux incident response workflow to isolate a compromised VPS at the network layer, capture volatile evidence and stop lateral movement without powering the server off.

  • Advanced
  • Linux VPS
  • 7 min read

By RunRacks Team · Published

Linux incident response starts with isolation: you cut the compromised server off from everything except your own admin session, so the attacker cannot move to other hosts, exfiltrate data or pull new payloads. This guide shows how to do that on a RunRacks Linux VPS without rebooting or powering the machine off, which keeps memory, running processes and open sockets available for triage.

Linux incident response: order of operations

Work in this order and do not skip ahead:

  1. Get an out-of-band console so a firewall mistake cannot lock you out.
  2. Capture volatile evidence (processes, sockets, logged-in users, routes).
  3. Block outbound traffic, then inbound traffic, allowing only your admin IP.
  4. Deal with the malicious processes.
  5. Preserve the disk and rotate every credential the host could have touched.

Rebooting or reinstalling the OS destroys most of what you need. Do it last, not first.

Before you start

  • Root or sudo access to the VPS, and the IP address you will administer from. Examples here use 203.0.113.10 and the user deploy.
  • A second, clean machine for searching, note-taking and password resets.
  • The Dashboard open at /dashboard in a separate browser tab. The browser console there works even when SSH is broken, and it is your way back in if firewall rules go wrong.
  • A rough idea of what the server is supposed to do, so you can tell normal traffic from attacker traffic.

Step 1: Record the session

Start a transcript so your notes are defensible later. On the console or over SSH:

Terminal
mkdir -p /root/ir
script -q -a /root/ir/session-$(date -u +%Y%m%dT%H%M%SZ).log

Then log the basics:

Terminal
date -u
hostname
uptime
who -a
w
last -n 20

last often shows the first sign of trouble: a login from a country you have never used, or a session that is still open.

Step 2: Capture volatile evidence

Do this before you change a single rule, because killing a process or dropping a connection destroys the evidence.

Terminal
ss -tunap > /root/ir/sockets.txt
ps auxf > /root/ir/processes.txt
lsof -nP -i > /root/ir/lsof-net.txt
ip addr > /root/ir/ip.txt
ip route > /root/ir/route.txt
arp -an > /root/ir/arp.txt
systemctl list-units --type=service --state=running > /root/ir/services.txt
journalctl --since "-24 hours" --no-pager > /root/ir/journal.txt

Also grab the scheduled-task surface, which is where persistence usually hides:

Terminal
crontab -l > /root/ir/crontab-root.txt 2>&1
ls -la /etc/cron.d /etc/cron.daily /etc/cron.hourly > /root/ir/cron-dirs.txt 2>&1
cat /etc/rc.local 2>/dev/null
cat /etc/ld.so.preload 2>/dev/null

Step 3: Block outbound traffic first

Outbound is what lets an attacker exfiltrate data, download a second-stage payload or reach your other servers. Cut it before inbound, and allow only your admin IP so your own SSH session survives.

Terminal
iptables -N IR_OUT
iptables -I OUTPUT 1 -j IR_OUT
iptables -A IR_OUT -o lo -j ACCEPT
iptables -A IR_OUT -d 203.0.113.10 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A IR_OUT -j DROP

Everything else, including DNS and package updates, now fails. That is expected and intentional. On nftables systems the same idea looks like this:

Terminal
nft add table inet ir
nft add chain inet ir out '{ type filter hook output priority 0; policy drop; }'
nft add rule inet ir out oif lo accept
nft add rule inet ir out ip daddr 203.0.113.10 ct state established,related accept

If you run firewalld on AlmaLinux or Rocky Linux, add the equivalent rich rules instead of mixing tools — see Configure firewalld on AlmaLinux and Rocky Linux.

Step 4: Restrict inbound traffic

Now lock the front door. Allow loopback and your admin IP on the SSH port only, then drop the rest.

Terminal
iptables -N IR_IN
iptables -I INPUT 1 -j IR_IN
iptables -A IR_IN -i lo -j ACCEPT
iptables -A IR_IN -s 203.0.113.10 -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A IR_IN -j DROP

If you moved SSH to a non-standard port earlier, use that port number instead — the procedure is the same as in Change the SSH port safely. If you have a long list of hostile addresses to block, ipset for high-performance IP blacklisting is far more efficient than hundreds of individual rules.

Step 5: Handle the malicious processes

With the network contained, you can look at what is running without it calling home. Identify the process, check what binary it is really running, then stop it.

Terminal
ps -eo pid,ppid,user,lstart,cmd --sort=start_time | tail -30
ls -l /proc/<PID>/exe
ls -l /proc/<PID>/cwd
cat /proc/<PID>/cmdline | tr '\0' ' '

Then stop and disable it so it does not come back:

Terminal
systemctl disable --now suspicious-unit.service
kill -9 <PID>

Step 6: Preserve the disk and rotate credentials

Do not wipe the server yet. Check the Dashboard at /dashboard for any snapshot or reinstall option before you act — an OS reinstall erases the disk completely, so it is the last step, not a fix.

From your clean machine, rotate everything the host could have read:

  • SSH keys in /root/.ssh/authorized_keys and every user's ~/.ssh/authorized_keys.
  • Passwords for databases, panels, mail accounts and any API tokens stored on disk.
  • Cloud or DNS credentials, and any deploy keys used by CI.

Then rebuild the VPS from a known-good image and restore only data you have verified.

Check it works

Confirm the isolation is actually in place:

Terminal
ss -tunap
iptables -L IR_OUT -n -v
iptables -L IR_IN -n -v

You should see no established connections to unfamiliar addresses, and rising packet counters on the DROP rules. From another host, a port check against a public service should time out:

Terminal
nc -zv 203.0.113.10 80

From the server itself, outbound should fail too:

Terminal
curl -m 5 https://example.com

Your own SSH session from 203.0.113.10 must still work. If it does, isolation is complete.

Troubleshooting

  • You lost SSH access. Cause: the inbound rule dropped your session, usually because your source IP changed. Fix: open the browser console in the Dashboard and run iptables -D INPUT -j IR_IN or iptables -F IR_IN.
  • DNS and package installs fail. Cause: outbound is dropped by design. Fix: temporarily add an allow rule for your resolver IP, or accept the breakage and work from your clean machine.
  • Customer-facing services are unreachable. Cause: inbound isolation. Fix: expected during an incident. Document the outage window and restore services after the rebuild.
  • The malicious process restarts immediately. Cause: persistence via systemd, cron, /etc/rc.local or ld.so.preload. Fix: re-check the files captured in Step 2 and remove the unit or entry, not just the process.
  • Rules disappear after a reboot. Cause: they were never persisted. Fix: only persist once you are certain of your admin IP, then verify access from a second network before you log out.

Next steps

Was this guide helpful?

Still stuck?

Ask RunRacks AI for a quick answer, or open a ticket and our team will take a look at your server.

Security

Check the Windows Firewall rule for RDP port 12230

Find and verify the inbound TCP 12230 rule that lets Remote Desktop reach your RunRacks server, recreate it if it was deleted and scope it without locking yourself out.

  • Intermediate
  • Windows VPS · Admin RDP
  • 5 min read

Security

iptables basics: list, allow, drop and save rules

Learn how iptables chains and rule order work, then build a small ruleset that allows SSH and web traffic, drops everything else and can be edited by line number.

  • Intermediate
  • Linux VPS
  • 6 min read