Malware Scanning on Linux: Rootkit Detection and System Integrity Checks

Rootkit detection on Linux means comparing what your system claims to be against what it actually is: binaries, kernel modules, open ports and file contents. In this guide you install rkhunter and chkrootkit for signature-based rootkit scanning, and AIDE for file-level system integrity checks, so a hidden backdoor or modified binary shows up instead of sitting quietly on the VPS.

A rootkit's whole purpose is to hide from normal inspection. ps, ls and netstat may be lying to you if the binaries behind them were replaced. These tools work from a different angle: hashes, known-bad signatures and intrusion indicators that are harder to fake without also breaking the system.

Before you start

You need:

• A Linux VPS running Ubuntu 22.04/24.04, Debian 12, or AlmaLinux/Rocky 8–9, with sudo or root access.
• SSH access. If you have not connected yet, see Connect to your Linux VPS with SSH.
• Roughly 1 GB of free disk for the AIDE database and logs. Check with df -h /.
• A mail address if you want scan warnings delivered, e.g. root@example.com.

Install the tools. On Debian and Ubuntu:

sudo apt update
sudo apt install -y rkhunter chkrootkit aide

On AlmaLinux and Rocky Linux:

sudo dnf install -y epel-release
sudo dnf install -y rkhunter aide

chkrootkit is not always packaged for RHEL-family systems. Check with dnf list chkrootkit; if it is missing, skip it and rely on rkhunter plus AIDE, which still gives you two independent detection paths.

Confirm the versions:

rkhunter --version | head -n 1
chkrootkit -V
aide --version | head -n 1

Build a clean baseline

rkhunter compares current file hashes against a stored baseline. Create or refresh that baseline while you trust the system:

sudo rkhunter --update
sudo rkhunter --propupd

--update refreshes the signature data files; --propupd writes the current file properties to /var/lib/rkhunter/db/rkhunter.dat. On a fresh install you can also populate package metadata so rkhunter can spot binaries that no package owns:

sudo rkhunter --propupd --pkgmgr DEB   # Debian/Ubuntu
sudo rkhunter --propupd --pkgmgr RPM   # AlmaLinux/Rocky

Then run the first full check:

sudo rkhunter --check --skip-keypress

The summary appears on screen and the full detail goes to /var/log/rkhunter.log. Expect a handful of warnings on a healthy server — see Troubleshooting for the common ones.

Now run chkrootkit as an independent second opinion:

sudo chkrootkit -q

-q prints only positive findings, so quiet output is good news. For the detailed view per test, use sudo chkrootkit -x | less.

Configure rkhunter for scheduled rootkit detection

Open the config file:

sudo nano /etc/rkhunter.conf

Set at least these values (uncomment or adjust the existing lines):

UPDATE_MIRRORS=1
MIRRORS_MODE=0
MAIL-ON-WARNING=root@example.com
MAIL_CMD=mail -s "[rkhunter] $(hostname)"

• UPDATE_MIRRORS=1 lets rkhunter --update pull fresh signature files.
• MAIL-ON-WARNING sends an email when a check produces warnings, so you do not have to remember to read the log.
• If scans feel slow on a small VPS, disable the heaviest optional test with DISABLE_TESTS="suspscan" rather than turning off file or rootkit checks.

Do not silence a warning with ALLOW_* entries until you have confirmed the file is legitimate. Whitelisting is the step that hides a real compromise.

Run the scheduled job as a cron entry:

sudo tee /etc/cron.d/rkhunter-scan >/dev/null <<'EOF'
0 3 * * * root /usr/bin/rkhunter --cronjob --update --quiet
EOF
sudo chmod 644 /etc/cron.d/rkhunter-scan

--cronjob implies --skip-keypress and suppresses interactive prompts. Output lands in /var/log/rkhunter.log.

Add file integrity monitoring with AIDE

rkhunter knows about known rootkits; AIDE knows about change. It records a hash, size, permission and timestamp snapshot of the files you choose, then reports anything that differs. That catches a replaced binary or a new file in /etc even when no signature exists yet.

Initialise the database on Ubuntu or Debian:

sudo aideinit

This takes a few minutes and creates /var/lib/aide/aide.db. On AlmaLinux and Rocky:

sudo aide --init
sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

Run a check. Debian and Ubuntu use the wrapper so the correct config and database are passed:

sudo aide.wrapper --check

On RHEL-family systems:

sudo aide --check

A clean run prints something like AIDE found NO differences between database and filesystem. Any added, removed or changed entries are listed per file — investigate each one before touching the database.

After you apply package updates on purpose, refresh both baselines so the next scan is meaningful:

sudo rkhunter --propupd
sudo aide.wrapper --update && sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

On RHEL-family: sudo aide --update then sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz.

To automate the AIDE run, a systemd timer keeps it in one place — see Automating server restarts and health checks with systemd services for the pattern.

Check it works

Trigger a harmless change so you know AIDE is actually watching:

echo "test $(date)" | sudo tee -a /etc/aide-test-marker
sudo aide.wrapper --check | grep -i aide-test-marker

You should see the new file reported as added. Remove it and rebuild the baseline:

sudo rm /etc/aide-test-marker
sudo aide.wrapper --update && sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

For rkhunter, confirm the baseline file exists and the last scan wrote results:

ls -lh /var/lib/rkhunter/db/rkhunter.dat
sudo grep -c "Warning" /var/log/rkhunter.log

A non-zero warning count is normal after apt upgrade or dnf update — the point is that you review each one.

Troubleshooting

• rkhunter warns "The file properties have changed" for common binaries. Cause: you applied package updates but did not refresh the baseline. Compare the file against its package with dpkg -V coreutils or rpm -V coreutils, then run sudo rkhunter --propupd.
• "Package manager" or "PKGMGR" check skipped. Cause: no package database tool found, or the package manager type was not detected. Re-run sudo rkhunter --propupd --pkgmgr DEB (or RPM), and install debsums on Debian-based systems.
• chkrootkit prints "INFECTED" on a clean server. Cause: well-known false positives such as the LKM or bindshell tests on kernels with unusual module handling. Confirm with ps aux, ss -tulpn and an AIDE run; if all three agree, it is a false positive. Check the chkrootkit documentation for the current list, as entries change between releases.
• AIDE reports hundreds of changes after apt upgrade. Cause: expected — package files, hashes and timestamps all moved. Review the list for changes outside /usr and /var/lib/dpkg, then rebuild the database.
• Scans slow the server badly. Cause: disk I/O on a small VPS while hashing thousands of files. Run scans off-peak and lower the priority: sudo nice -n 19 ionice -c3 rkhunter --check --skip-keypress.

Next steps

• Pair these scans with antivirus coverage for uploaded files: Malware Scanning on Linux: Configuring ClamAV for Web Root Protection.
• Lock down the binaries these tools hash: Linux Hardening: Securing System Binaries with SELinux or AppArmor.
• Catch changes the moment they happen, not at 03:00: Using inotify-tools to Monitor File System Changes in Real-Time on Linux.
• If a scan finds something real, stop and read Incident Response: Preserving Forensic Evidence from a Compromised Linux Server before deleting anything.