Linux Incident Response: Isolate a Compromised Server Without Downtime

Linux incident response starts with isolation: you cut the compromised server off from everything except your own admin session, so the attacker cannot move to other hosts, exfiltrate data or pull new payloads. This guide shows how to do that on a Linux VPS without rebooting or powering the machine off, which keeps memory, running processes and open sockets available for triage.

Linux incident response: order of operations

Work in this order and do not skip ahead:

1. Get an out-of-band console so a firewall mistake cannot lock you out.
2. Capture volatile evidence (processes, sockets, logged-in users, routes).
3. Block outbound traffic, then inbound traffic, allowing only your admin IP.
4. Deal with the malicious processes.
5. Preserve the disk and rotate every credential the host could have touched.

Rebooting or reinstalling the OS destroys most of what you need. Do it last, not first.

Before you start

• Root or sudo access to the VPS, and the IP address you will administer from. Examples here use 203.0.113.10 and the user deploy.
• A second, clean machine for searching, note-taking and password resets.
• Your VPS provider's control panel open at your VPS provider's control panel in a separate browser tab. The browser console there works even when SSH is broken, and it is your way back in if firewall rules go wrong.
• A rough idea of what the server is supposed to do, so you can tell normal traffic from attacker traffic.

Step 1: Record the session

Start a transcript so your notes are defensible later. On the console or over SSH:

mkdir -p /root/ir
script -q -a /root/ir/session-$(date -u +%Y%m%dT%H%M%SZ).log

Then log the basics:

date -u
hostname
uptime
who -a
w
last -n 20

last often shows the first sign of trouble: a login from a country you have never used, or a session that is still open.

Step 2: Capture volatile evidence

Do this before you change a single rule, because killing a process or dropping a connection destroys the evidence.

ss -tunap > /root/ir/sockets.txt
ps auxf > /root/ir/processes.txt
lsof -nP -i > /root/ir/lsof-net.txt
ip addr > /root/ir/ip.txt
ip route > /root/ir/route.txt
arp -an > /root/ir/arp.txt
systemctl list-units --type=service --state=running > /root/ir/services.txt
journalctl --since "-24 hours" --no-pager > /root/ir/journal.txt

Also grab the scheduled-task surface, which is where persistence usually hides:

crontab -l > /root/ir/crontab-root.txt 2>&1
ls -la /etc/cron.d /etc/cron.daily /etc/cron.hourly > /root/ir/cron-dirs.txt 2>&1
cat /etc/rc.local 2>/dev/null
cat /etc/ld.so.preload 2>/dev/null

Step 3: Block outbound traffic first

Outbound is what lets an attacker exfiltrate data, download a second-stage payload or reach your other servers. Cut it before inbound, and allow only your admin IP so your own SSH session survives.

iptables -N IR_OUT
iptables -I OUTPUT 1 -j IR_OUT
iptables -A IR_OUT -o lo -j ACCEPT
iptables -A IR_OUT -d 203.0.113.10 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A IR_OUT -j DROP

Everything else, including DNS and package updates, now fails. That is expected and intentional. On nftables systems the same idea looks like this:

nft add table inet ir
nft add chain inet ir out '{ type filter hook output priority 0; policy drop; }'
nft add rule inet ir out oif lo accept
nft add rule inet ir out ip daddr 203.0.113.10 ct state established,related accept

If you run firewalld on AlmaLinux or Rocky Linux, add the equivalent rich rules instead of mixing tools — see Configure firewalld on AlmaLinux and Rocky Linux.

Step 4: Restrict inbound traffic

Now lock the front door. Allow loopback and your admin IP on the SSH port only, then drop the rest.

iptables -N IR_IN
iptables -I INPUT 1 -j IR_IN
iptables -A IR_IN -i lo -j ACCEPT
iptables -A IR_IN -s 203.0.113.10 -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A IR_IN -j DROP

If you moved SSH to a non-standard port earlier, use that port number instead — the procedure is the same as in Change the SSH port safely. If you have a long list of hostile addresses to block, ipset for high-performance IP blacklisting is far more efficient than hundreds of individual rules.

Step 5: Handle the malicious processes

With the network contained, you can look at what is running without it calling home. Identify the process, check what binary it is really running, then stop it.

ps -eo pid,ppid,user,lstart,cmd --sort=start_time | tail -30
ls -l /proc/<PID>/exe
ls -l /proc/<PID>/cwd
cat /proc/<PID>/cmdline | tr '\0' ' '

Then stop and disable it so it does not come back:

systemctl disable --now suspicious-unit.service
kill -9 <PID>

Step 6: Preserve the disk and rotate credentials

Do not wipe the server yet. Check your VPS provider's control panel for any snapshot or reinstall option before you act — an OS reinstall erases the disk completely, so it is the last step, not a fix.

From your clean machine, rotate everything the host could have read:

• SSH keys in /root/.ssh/authorized_keys and every user's ~/.ssh/authorized_keys.
• Passwords for databases, panels, mail accounts and any API tokens stored on disk.
• Cloud or DNS credentials, and any deploy keys used by CI.

Then rebuild the VPS from a known-good image and restore only data you have verified.

Check it works

Confirm the isolation is actually in place:

ss -tunap
iptables -L IR_OUT -n -v
iptables -L IR_IN -n -v

You should see no established connections to unfamiliar addresses, and rising packet counters on the DROP rules. From another host, a port check against a public service should time out:

nc -zv 203.0.113.10 80

From the server itself, outbound should fail too:

curl -m 5 https://example.com

Your own SSH session from 203.0.113.10 must still work. If it does, isolation is complete.

Troubleshooting

• You lost SSH access. Cause: the inbound rule dropped your session, usually because your source IP changed. Fix: open the browser console in your VPS provider's control panel and run iptables -D INPUT -j IR_IN or iptables -F IR_IN.
• DNS and package installs fail. Cause: outbound is dropped by design. Fix: temporarily add an allow rule for your resolver IP, or accept the breakage and work from your clean machine.
• Customer-facing services are unreachable. Cause: inbound isolation. Fix: expected during an incident. Document the outage window and restore services after the rebuild.
• The malicious process restarts immediately. Cause: persistence via systemd, cron, /etc/rc.local or ld.so.preload. Fix: re-check the files captured in Step 2 and remove the unit or entry, not just the process.
• Rules disappear after a reboot. Cause: they were never persisted. Fix: only persist once you are certain of your admin IP, then verify access from a second network before you log out.

Next steps

• Harden the rebuilt host with First steps after your Linux VPS is ready.
• Tighten remote access using Disable weak SSH ciphers and MAC algorithms.
• Set up off-host copies with Back up your Linux VPS with rsync, restic and cron.