Incident Response: Preserving Forensic Evidence from a Compromised Linux Server
Capture memory, network state and logs from a compromised Linux VPS before anything changes, then image the disk and record a chain of custody.
- Advanced
- Linux VPS
- 5 min read
Preserving forensic evidence on a Linux server means capturing volatile data — memory, network connections and running processes — before you touch the disk, then imaging the disk and copying logs without changing them. If you reboot, patch or clean the machine first, most of that evidence is gone. This guide walks through the order of capture, the exact commands, and how to keep a chain of custody that holds up to later review.
Before you start
You need:
- Root or sudo access to the affected server.
- A second host or a separate volume with enough free space for a full disk image.
- The browser console in the Dashboard as a fallback if SSH dies mid-capture.
- Static forensic binaries (dc3dd, AVML, a LiME module matching your kernel) downloaded to the evidence host, not to the suspect server.
Do not reboot, do not run apt upgrade or dnf update, and do not kill suspicious processes yet. Each of those actions destroys volatile evidence. Isolate the network instead — see Linux Incident Response: Isolate a Compromised Server Without Downtime.
Installing forensic tools on the compromised server writes to disk and changes timestamps. Prefer static binaries copied over SSH from your workstation, or run them from a read-only mount.
The order of volatility when preserving forensic evidence
Capture in this order, fastest-changing first:
- CPU registers and memory.
- Network connections, ARP cache and routing table.
- Running processes and open files.
- Temporary files and
/procentries. - Disk contents.
- Logs and archived data.
Anything you skip at step 1 may already be gone by step 3.
Capture volatile data
Create a case directory and record everything with UTC timestamps.
export CASE=/mnt/evidence/case-$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$CASE"
date -u > "$CASE/date.txt"
uptime > "$CASE/uptime.txt"
who -a > "$CASE/who.txt"
last -Faiw > "$CASE/last.txt"
ps auxwwf > "$CASE/ps.txt"
ss -tulpn > "$CASE/ss.txt"
lsof -nPi > "$CASE/lsof.txt"
ip addr show > "$CASE/ip.txt"
ip route show > "$CASE/route.txt"
ip neigh show > "$CASE/arp.txt"Also grab deleted files still held open by a process — a common sign of a dropped binary:
lsof +L1 > "$CASE/deleted-open-files.txt"For deeper connection analysis, see Using netstat and ss for detailed network connection analysis on Linux.
Capture memory
Memory holds decrypted payloads, keys and process arguments that never reach disk. Two tools work well on a VPS.
AVML is a static binary, so there is nothing to compile:
sudo ./avml /mnt/evidence/memory.avmlLiME needs a module built for the running kernel:
sudo insmod ./lime-$(uname -r).ko "path=/mnt/evidence/memory.lime format=lime"Loading a kernel module changes the running system. If the choice is between losing memory and altering the box slightly, capture the memory and document exactly what you loaded.
Hash the image as soon as it is written:
sha256sum /mnt/evidence/memory.* | tee -a "$CASE/hashes.txt"Image the disk
Write the image to external storage or stream it over the network — never to the same disk you are imaging.
sudo dc3dd if=/dev/vda of=/mnt/evidence/disk.dd hash=sha256 log=/mnt/evidence/dc3dd.log bs=1MIf local space is tight, stream it to your evidence host:
sudo dc3dd if=/dev/vda hash=sha256 bs=1M | nc 203.0.113.10 9000On the receiving host:
nc -l -p 9000 > disk.ddA live image is not byte-consistent — files change while you read them. Record the start and end time and state that in your report. If the server can be taken offline, image it from a rescue environment instead.
Preserve logs and file metadata
Copy the journal and log directories with permissions and extended attributes intact:
journalctl --no-pager -o short-iso > "$CASE/journal.txt"
tar --acls --xattrs -czf "$CASE/var-log.tar.gz" /var/logThen record metadata for recently modified files — timestamps are often the quickest route to the entry point:
find / -xdev -newermt "2026-01-01" -printf '%T+ %u %g %m %p\n' > "$CASE/recent-files.txt"Check the usual authentication files too: /var/log/auth.log or /var/log/secure, /var/log/wtmp, /var/log/btmp and /var/log/lastlog.
Record hashes and chain of custody
Every artefact needs a hash plus a note of who collected it, when, and with which tool version.
cd "$CASE" && sha256sum * > SHA256SUMS.txtWrite a short README.txt in the case directory with the server IP, hostname, collection window, tool versions and the name of the person collecting. Store the evidence on media you can write-protect afterwards.
Check it works
Verify the hashes before you copy anything off the evidence host:
sha256sum -c SHA256SUMS.txtThen confirm the artefacts are usable:
- The memory file is roughly the size of the VM's RAM.
disk.ddmounts read-only and lists a filesystem:sudo mount -o ro,loop disk.dd /mnt/image.journal.txtis not empty and covers the incident window.var-log.tar.gzextracts without errors.
Troubleshooting
- `dc3dd` stops with "No space left on device" — the destination is smaller than the disk. Stream the image over the network with
nc, or image only the partitions you need. - `insmod: ERROR: could not insert module` — the LiME module was built for a different kernel. Check
uname -r, rebuild against matching headers, or switch to AVML. - `journalctl` returns almost nothing — journald is running in volatile mode and logs live in
/run/log/journal. Copy that directory immediately; it disappears on reboot. - Commands hang or return odd output — a rootkit may be intercepting them. Work from the browser console in the Dashboard and run binaries from a static toolkit on read-only media.
- Timestamps on your evidence files keep changing — you are writing to a filesystem with atime enabled. Mount the evidence volume with
noatime, or write straight to the remote host.
Next steps
- Linux Incident Response: Isolate a Compromised Server Without Downtime — cut the attacker off before you collect.
- Using netstat and ss for detailed network connection analysis on Linux — read the connection data you just captured.
- Malware Scanning on Linux: Configuring ClamAV for Web Root Protection — scan the image once it is safely copied.
- Linux VPS hosting — rebuild on a clean server when the investigation is finished.
Was this guide helpful?
Still stuck?
Ask RunRacks AI for a quick answer, or open a ticket and our team will take a look at your server.
Related guides
Security
Check the Windows Firewall rule for RDP port 12230
Find and verify the inbound TCP 12230 rule that lets Remote Desktop reach your RunRacks server, recreate it if it was deleted and scope it without locking yourself out.
- Intermediate
- Windows VPS · Admin RDP
- 5 min read
Security
Linux Security: Using aide for File Integrity Monitoring on VPS
AIDE builds a checksum database of your system files and reports anything that changed. This guide installs it, creates a baseline, tunes the rules and schedules daily checks on a Linux VPS.
- Advanced
- Linux VPS
- 6 min read
Security
iptables basics: list, allow, drop and save rules
Learn how iptables chains and rule order work, then build a small ruleset that allows SSH and web traffic, drops everything else and can be edited by line number.
- Intermediate
- Linux VPS
- 6 min read