Malware Scanning on Linux: Rootkit Detection and System Integrity Checks

Install rkhunter, chkrootkit and AIDE on your Linux VPS to detect hidden rootkits, trojaned binaries and unauthorised file changes before they spread.

  • Intermediate
  • Linux VPS
  • 7 min read

By RunRacks Team · Published

Rootkit detection on Linux means comparing what your system claims to be against what it actually is: binaries, kernel modules, open ports and file contents. In this guide you install rkhunter and chkrootkit for signature-based rootkit scanning, and AIDE for file-level system integrity checks, so a hidden backdoor or modified binary shows up instead of sitting quietly on the VPS.

A rootkit's whole purpose is to hide from normal inspection. ps, ls and netstat may be lying to you if the binaries behind them were replaced. These tools work from a different angle: hashes, known-bad signatures and intrusion indicators that are harder to fake without also breaking the system.

Before you start

You need:

  • A Linux VPS running Ubuntu 22.04/24.04, Debian 12, or AlmaLinux/Rocky 8–9, with sudo or root access.
  • SSH access. If you have not connected yet, see Connect to your Linux VPS with SSH.
  • Roughly 1 GB of free disk for the AIDE database and logs. Check with df -h /.
  • A mail address if you want scan warnings delivered, e.g. [email protected].

Install the tools. On Debian and Ubuntu:

Terminal
sudo apt update
sudo apt install -y rkhunter chkrootkit aide

On AlmaLinux and Rocky Linux:

Terminal
sudo dnf install -y epel-release
sudo dnf install -y rkhunter aide

chkrootkit is not always packaged for RHEL-family systems. Check with dnf list chkrootkit; if it is missing, skip it and rely on rkhunter plus AIDE, which still gives you two independent detection paths.

Confirm the versions:

Terminal
rkhunter --version | head -n 1
chkrootkit -V
aide --version | head -n 1

Build a clean baseline

rkhunter compares current file hashes against a stored baseline. Create or refresh that baseline while you trust the system:

Terminal
sudo rkhunter --update
sudo rkhunter --propupd

--update refreshes the signature data files; --propupd writes the current file properties to /var/lib/rkhunter/db/rkhunter.dat. On a fresh install you can also populate package metadata so rkhunter can spot binaries that no package owns:

Terminal
sudo rkhunter --propupd --pkgmgr DEB   # Debian/Ubuntu
sudo rkhunter --propupd --pkgmgr RPM   # AlmaLinux/Rocky

Then run the first full check:

Terminal
sudo rkhunter --check --skip-keypress

The summary appears on screen and the full detail goes to /var/log/rkhunter.log. Expect a handful of warnings on a healthy server — see Troubleshooting for the common ones.

Now run chkrootkit as an independent second opinion:

Terminal
sudo chkrootkit -q

-q prints only positive findings, so quiet output is good news. For the detailed view per test, use sudo chkrootkit -x | less.

Configure rkhunter for scheduled rootkit detection

Open the config file:

Terminal
sudo nano /etc/rkhunter.conf

Set at least these values (uncomment or adjust the existing lines):

INI
UPDATE_MIRRORS=1
MIRRORS_MODE=0
[email protected]
MAIL_CMD=mail -s "[rkhunter] $(hostname)"
  • UPDATE_MIRRORS=1 lets rkhunter --update pull fresh signature files.
  • MAIL-ON-WARNING sends an email when a check produces warnings, so you do not have to remember to read the log.
  • If scans feel slow on a small VPS, disable the heaviest optional test with DISABLE_TESTS="suspscan" rather than turning off file or rootkit checks.

Do not silence a warning with ALLOW_* entries until you have confirmed the file is legitimate. Whitelisting is the step that hides a real compromise.

Run the scheduled job as a cron entry:

Terminal
sudo tee /etc/cron.d/rkhunter-scan >/dev/null <<'EOF'
0 3 * * * root /usr/bin/rkhunter --cronjob --update --quiet
EOF
sudo chmod 644 /etc/cron.d/rkhunter-scan

--cronjob implies --skip-keypress and suppresses interactive prompts. Output lands in /var/log/rkhunter.log.

Add file integrity monitoring with AIDE

rkhunter knows about known rootkits; AIDE knows about change. It records a hash, size, permission and timestamp snapshot of the files you choose, then reports anything that differs. That catches a replaced binary or a new file in /etc even when no signature exists yet.

Initialise the database on Ubuntu or Debian:

Terminal
sudo aideinit

This takes a few minutes and creates /var/lib/aide/aide.db. On AlmaLinux and Rocky:

Terminal
sudo aide --init
sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

Run a check. Debian and Ubuntu use the wrapper so the correct config and database are passed:

Terminal
sudo aide.wrapper --check

On RHEL-family systems:

Terminal
sudo aide --check

A clean run prints something like AIDE found NO differences between database and filesystem. Any added, removed or changed entries are listed per file — investigate each one before touching the database.

After you apply package updates on purpose, refresh both baselines so the next scan is meaningful:

Terminal
sudo rkhunter --propupd
sudo aide.wrapper --update && sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

On RHEL-family: sudo aide --update then sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz.

To automate the AIDE run, a systemd timer keeps it in one place — see Automating server restarts and health checks with systemd services for the pattern.

Check it works

Trigger a harmless change so you know AIDE is actually watching:

Terminal
echo "test $(date)" | sudo tee -a /etc/aide-test-marker
sudo aide.wrapper --check | grep -i aide-test-marker

You should see the new file reported as added. Remove it and rebuild the baseline:

Terminal
sudo rm /etc/aide-test-marker
sudo aide.wrapper --update && sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

For rkhunter, confirm the baseline file exists and the last scan wrote results:

Terminal
ls -lh /var/lib/rkhunter/db/rkhunter.dat
sudo grep -c "Warning" /var/log/rkhunter.log

A non-zero warning count is normal after apt upgrade or dnf update — the point is that you review each one.

Troubleshooting

  • rkhunter warns "The file properties have changed" for common binaries. Cause: you applied package updates but did not refresh the baseline. Compare the file against its package with dpkg -V coreutils or rpm -V coreutils, then run sudo rkhunter --propupd.
  • "Package manager" or "PKGMGR" check skipped. Cause: no package database tool found, or the package manager type was not detected. Re-run sudo rkhunter --propupd --pkgmgr DEB (or RPM), and install debsums on Debian-based systems.
  • chkrootkit prints "INFECTED" on a clean server. Cause: well-known false positives such as the LKM or bindshell tests on kernels with unusual module handling. Confirm with ps aux, ss -tulpn and an AIDE run; if all three agree, it is a false positive. Check the chkrootkit documentation for the current list, as entries change between releases.
  • AIDE reports hundreds of changes after `apt upgrade`. Cause: expected — package files, hashes and timestamps all moved. Review the list for changes outside /usr and /var/lib/dpkg, then rebuild the database.
  • Scans slow the server badly. Cause: disk I/O on a small VPS while hashing thousands of files. Run scans off-peak and lower the priority: sudo nice -n 19 ionice -c3 rkhunter --check --skip-keypress.

Next steps

Was this guide helpful?

Still stuck?

Ask RunRacks AI for a quick answer, or open a ticket and our team will take a look at your server.

Security

Check the Windows Firewall rule for RDP port 12230

Find and verify the inbound TCP 12230 rule that lets Remote Desktop reach your RunRacks server, recreate it if it was deleted and scope it without locking yourself out.

  • Intermediate
  • Windows VPS · Admin RDP
  • 5 min read

Security

iptables basics: list, allow, drop and save rules

Learn how iptables chains and rule order work, then build a small ruleset that allows SSH and web traffic, drops everything else and can be edited by line number.

  • Intermediate
  • Linux VPS
  • 6 min read