Incident Response: Preserving Forensic Evidence from a Compromised Linux Server

Capture memory, network state and logs from a compromised Linux VPS before anything changes, then image the disk and record a chain of custody.

  • Advanced
  • Linux VPS
  • 5 min read

By RunRacks Team · Published

Preserving forensic evidence on a Linux server means capturing volatile data — memory, network connections and running processes — before you touch the disk, then imaging the disk and copying logs without changing them. If you reboot, patch or clean the machine first, most of that evidence is gone. This guide walks through the order of capture, the exact commands, and how to keep a chain of custody that holds up to later review.

Before you start

You need:

  • Root or sudo access to the affected server.
  • A second host or a separate volume with enough free space for a full disk image.
  • The browser console in the Dashboard as a fallback if SSH dies mid-capture.
  • Static forensic binaries (dc3dd, AVML, a LiME module matching your kernel) downloaded to the evidence host, not to the suspect server.

The order of volatility when preserving forensic evidence

Capture in this order, fastest-changing first:

  1. CPU registers and memory.
  2. Network connections, ARP cache and routing table.
  3. Running processes and open files.
  4. Temporary files and /proc entries.
  5. Disk contents.
  6. Logs and archived data.

Anything you skip at step 1 may already be gone by step 3.

Capture volatile data

Create a case directory and record everything with UTC timestamps.

Terminal
export CASE=/mnt/evidence/case-$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$CASE"
date -u > "$CASE/date.txt"
uptime > "$CASE/uptime.txt"
who -a > "$CASE/who.txt"
last -Faiw > "$CASE/last.txt"
ps auxwwf > "$CASE/ps.txt"
ss -tulpn > "$CASE/ss.txt"
lsof -nPi > "$CASE/lsof.txt"
ip addr show > "$CASE/ip.txt"
ip route show > "$CASE/route.txt"
ip neigh show > "$CASE/arp.txt"

Also grab deleted files still held open by a process — a common sign of a dropped binary:

Terminal
lsof +L1 > "$CASE/deleted-open-files.txt"

For deeper connection analysis, see Using netstat and ss for detailed network connection analysis on Linux.

Capture memory

Memory holds decrypted payloads, keys and process arguments that never reach disk. Two tools work well on a VPS.

AVML is a static binary, so there is nothing to compile:

Terminal
sudo ./avml /mnt/evidence/memory.avml

LiME needs a module built for the running kernel:

Terminal
sudo insmod ./lime-$(uname -r).ko "path=/mnt/evidence/memory.lime format=lime"

Hash the image as soon as it is written:

Terminal
sha256sum /mnt/evidence/memory.* | tee -a "$CASE/hashes.txt"

Image the disk

Write the image to external storage or stream it over the network — never to the same disk you are imaging.

Terminal
sudo dc3dd if=/dev/vda of=/mnt/evidence/disk.dd hash=sha256 log=/mnt/evidence/dc3dd.log bs=1M

If local space is tight, stream it to your evidence host:

Terminal
sudo dc3dd if=/dev/vda hash=sha256 bs=1M | nc 203.0.113.10 9000

On the receiving host:

Terminal
nc -l -p 9000 > disk.dd

Preserve logs and file metadata

Copy the journal and log directories with permissions and extended attributes intact:

Terminal
journalctl --no-pager -o short-iso > "$CASE/journal.txt"
tar --acls --xattrs -czf "$CASE/var-log.tar.gz" /var/log

Then record metadata for recently modified files — timestamps are often the quickest route to the entry point:

Terminal
find / -xdev -newermt "2026-01-01" -printf '%T+ %u %g %m %p\n' > "$CASE/recent-files.txt"

Check the usual authentication files too: /var/log/auth.log or /var/log/secure, /var/log/wtmp, /var/log/btmp and /var/log/lastlog.

Record hashes and chain of custody

Every artefact needs a hash plus a note of who collected it, when, and with which tool version.

Terminal
cd "$CASE" && sha256sum * > SHA256SUMS.txt

Write a short README.txt in the case directory with the server IP, hostname, collection window, tool versions and the name of the person collecting. Store the evidence on media you can write-protect afterwards.

Check it works

Verify the hashes before you copy anything off the evidence host:

Terminal
sha256sum -c SHA256SUMS.txt

Then confirm the artefacts are usable:

  • The memory file is roughly the size of the VM's RAM.
  • disk.dd mounts read-only and lists a filesystem: sudo mount -o ro,loop disk.dd /mnt/image.
  • journal.txt is not empty and covers the incident window.
  • var-log.tar.gz extracts without errors.

Troubleshooting

  • `dc3dd` stops with "No space left on device" — the destination is smaller than the disk. Stream the image over the network with nc, or image only the partitions you need.
  • `insmod: ERROR: could not insert module` — the LiME module was built for a different kernel. Check uname -r, rebuild against matching headers, or switch to AVML.
  • `journalctl` returns almost nothing — journald is running in volatile mode and logs live in /run/log/journal. Copy that directory immediately; it disappears on reboot.
  • Commands hang or return odd output — a rootkit may be intercepting them. Work from the browser console in the Dashboard and run binaries from a static toolkit on read-only media.
  • Timestamps on your evidence files keep changing — you are writing to a filesystem with atime enabled. Mount the evidence volume with noatime, or write straight to the remote host.

Next steps

Was this guide helpful?

Still stuck?

Ask RunRacks AI for a quick answer, or open a ticket and our team will take a look at your server.

Security

Check the Windows Firewall rule for RDP port 12230

Find and verify the inbound TCP 12230 rule that lets Remote Desktop reach your RunRacks server, recreate it if it was deleted and scope it without locking yourself out.

  • Intermediate
  • Windows VPS · Admin RDP
  • 5 min read

Security

iptables basics: list, allow, drop and save rules

Learn how iptables chains and rule order work, then build a small ruleset that allows SSH and web traffic, drops everything else and can be edited by line number.

  • Intermediate
  • Linux VPS
  • 6 min read