Linux Incident Response: Log Preservation and Analysis on a VPS

Linux incident response log preservation starts with copying /var/log/auth.log, syslog and the systemd journal off the server before anything else on it changes. This guide shows you how to capture those files, hash them so you can prove they were not edited, and read them to rebuild a timeline of what an intruder did.

Do this before you reboot, before you reinstall, and before you run any cleanup script. Logs are the only record of the intrusion that the attacker cannot rewrite once they are in your hands.

Before you start

You need:

• SSH access with a sudo-capable account (see create a sudo user).
• A second machine to hold the evidence — your laptop or another VPS you control. Do not keep the only copy on the compromised server.
• Enough free space for the logs. Check with df -h /var/log.
• The server's timezone and clock state, so timestamps line up later.

Record the basics first

Capture the context that makes the logs readable later.

export CASE=/root/ir-$(date -u +%Y%m%dT%H%M%SZ)
sudo mkdir -p "$CASE"
{
  date -u
  timedatectl
  hostname
  uname -a
  uptime
  who -a
  last -Faiw | head -40
} | sudo tee "$CASE/00-context.txt"

The CASE variable keeps every later command short. Open a second SSH session and leave it open — if the attacker kills your first session, you still have a shell.

Linux incident response log preservation

Copy, do not move. cp -a keeps timestamps and permissions, which matters if you later need to show file metadata.

sudo cp -a /var/log/auth.log*   "$CASE"/ 2>/dev/null
sudo cp -a /var/log/syslog*     "$CASE"/ 2>/dev/null
sudo cp -a /var/log/secure*     "$CASE"/ 2>/dev/null
sudo cp -a /var/log/messages*   "$CASE"/ 2>/dev/null
sudo cp -a /var/log/wtmp* /var/log/btmp* /var/log/lastlog "$CASE"/ 2>/dev/null
sudo cp -a /etc/passwd /etc/shadow /etc/group /etc/sudoers "$CASE"/ 2>/dev/null

Debian and Ubuntu write authentication events to auth.log and general events to syslog. AlmaLinux and Rocky Linux use secure and messages instead. Copying both sets costs nothing and covers a distribution you misremember.

Then pull the journal, which often holds events rsyslog never wrote to disk:

sudo journalctl --no-pager > "$CASE/journal-all.txt"
sudo journalctl -b -1 --no-pager > "$CASE/journal-previous-boot.txt" 2>/dev/null
sudo journalctl -u ssh -u sshd --no-pager > "$CASE/journal-ssh.txt"

-b -1 reads the previous boot. It only works if persistent journal storage is enabled — if the file is empty, this host keeps its journal in /run/log/journal only.

Hash the evidence

A hash proves the copy has not changed since you took it. Generate one manifest for the whole case directory.

cd "$CASE"
sudo find . -type f ! -name SHA256SUMS -exec sha256sum {} + | sudo tee SHA256SUMS
sudo sha256sum SHA256SUMS | sudo tee SHA256SUMS.sha256

Store SHA256SUMS and SHA256SUMS.sha256 with the logs. Anyone you hand the evidence to can run sha256sum -c SHA256SUMS and confirm nothing was touched.

Move the copy off the server

Push the case directory to a host the attacker does not control.

sudo rsync -a --progress "$CASE/" deploy@backup.example.com:/evidence/$(hostname)/

If you have no second host, pull it from your workstation instead:

scp -r root@203.0.113.10:"$CASE" ./evidence/

Verify the hashes on the copy before you rely on it:

sha256sum -c SHA256SUMS

Run that inside the copied case directory. If every line reports OK, the evidence is intact and you can analyse it safely.