Incident Response: Preserving Forensic Evidence from a Compromised Linux Server

Preserving forensic evidence on a Linux server means capturing volatile data — memory, network connections and running processes — before you touch the disk, then imaging the disk and copying logs without changing them. If you reboot, patch or clean the machine first, most of that evidence is gone. This guide walks through the order of capture, the exact commands, and how to keep a chain of custody that holds up to later review.

Before you start

You need:

• Root or sudo access to the affected server.
• A second host or a separate volume with enough free space for a full disk image.
• The browser console in your VPS provider's control panel as a fallback if SSH dies mid-capture.
• Static forensic binaries (dc3dd, AVML, a LiME module matching your kernel) downloaded to the evidence host, not to the suspect server.

The order of volatility when preserving forensic evidence

Capture in this order, fastest-changing first:

1. CPU registers and memory.
2. Network connections, ARP cache and routing table.
3. Running processes and open files.
4. Temporary files and /proc entries.
5. Disk contents.
6. Logs and archived data.

Anything you skip at step 1 may already be gone by step 3.

Capture volatile data

Create a case directory and record everything with UTC timestamps.

export CASE=/mnt/evidence/case-$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$CASE"
date -u > "$CASE/date.txt"
uptime > "$CASE/uptime.txt"
who -a > "$CASE/who.txt"
last -Faiw > "$CASE/last.txt"
ps auxwwf > "$CASE/ps.txt"
ss -tulpn > "$CASE/ss.txt"
lsof -nPi > "$CASE/lsof.txt"
ip addr show > "$CASE/ip.txt"
ip route show > "$CASE/route.txt"
ip neigh show > "$CASE/arp.txt"

Also grab deleted files still held open by a process — a common sign of a dropped binary:

lsof +L1 > "$CASE/deleted-open-files.txt"

For deeper connection analysis, see Using netstat and ss for detailed network connection analysis on Linux.

Capture memory

Memory holds decrypted payloads, keys and process arguments that never reach disk. Two tools work well on a VPS.

AVML is a static binary, so there is nothing to compile:

sudo ./avml /mnt/evidence/memory.avml

LiME needs a module built for the running kernel:

sudo insmod ./lime-$(uname -r).ko "path=/mnt/evidence/memory.lime format=lime"

Hash the image as soon as it is written:

sha256sum /mnt/evidence/memory.* | tee -a "$CASE/hashes.txt"

Image the disk

Write the image to external storage or stream it over the network — never to the same disk you are imaging.

sudo dc3dd if=/dev/vda of=/mnt/evidence/disk.dd hash=sha256 log=/mnt/evidence/dc3dd.log bs=1M

If local space is tight, stream it to your evidence host:

sudo dc3dd if=/dev/vda hash=sha256 bs=1M | nc 203.0.113.10 9000

On the receiving host:

nc -l -p 9000 > disk.dd

Preserve logs and file metadata

Copy the journal and log directories with permissions and extended attributes intact:

journalctl --no-pager -o short-iso > "$CASE/journal.txt"
tar --acls --xattrs -czf "$CASE/var-log.tar.gz" /var/log

Then record metadata for recently modified files — timestamps are often the quickest route to the entry point:

find / -xdev -newermt "2026-01-01" -printf '%T+ %u %g %m %p\n' > "$CASE/recent-files.txt"

Check the usual authentication files too: /var/log/auth.log or /var/log/secure, /var/log/wtmp, /var/log/btmp and /var/log/lastlog.

Record hashes and chain of custody

Every artefact needs a hash plus a note of who collected it, when, and with which tool version.

cd "$CASE" && sha256sum * > SHA256SUMS.txt

Write a short README.txt in the case directory with the server IP, hostname, collection window, tool versions and the name of the person collecting. Store the evidence on media you can write-protect afterwards.

Check it works

Verify the hashes before you copy anything off the evidence host:

sha256sum -c SHA256SUMS.txt

Then confirm the artefacts are usable:

• The memory file is roughly the size of the VM's RAM.
• disk.dd mounts read-only and lists a filesystem: sudo mount -o ro,loop disk.dd /mnt/image.
• journal.txt is not empty and covers the incident window.
• var-log.tar.gz extracts without errors.

Troubleshooting

• dc3dd stops with "No space left on device" — the destination is smaller than the disk. Stream the image over the network with nc, or image only the partitions you need.
• insmod: ERROR: could not insert module — the LiME module was built for a different kernel. Check uname -r, rebuild against matching headers, or switch to AVML.
• journalctl returns almost nothing — journald is running in volatile mode and logs live in /run/log/journal. Copy that directory immediately; it disappears on reboot.
• Commands hang or return odd output — a rootkit may be intercepting them. Work from the browser console in your VPS provider's control panel and run binaries from a static toolkit on read-only media.
• Timestamps on your evidence files keep changing — you are writing to a filesystem with atime enabled. Mount the evidence volume with noatime, or write straight to the remote host.

Next steps

• Linux Incident Response: Isolate a Compromised Server Without Downtime — cut the attacker off before you collect.
• Using netstat and ss for detailed network connection analysis on Linux — read the connection data you just captured.
• Malware Scanning on Linux: Configuring ClamAV for Web Root Protection — scan the image once it is safely copied.