Security

Linux Update Strategy: Testing Security Patches Before Production

Build a Linux update strategy that tests security patches on a staging server first, then promotes them to production with a canary rollout and a rollback path.

AdvancedLinux server6 min read

A Linux update strategy that tests security patches before production keeps one bad kernel or library from taking down every server at once. This guide builds a staging tier, applies security and kernel updates there first, and promotes them to production only after your checks pass.

Before you start

You need:

  • A Linux VPS running Ubuntu 22.04/24.04, Debian 12, AlmaLinux 8–9 or Rocky 8–9, with root or a sudo user — see creating a sudo user if you haven't yet.
  • A second VPS to act as staging. Your VPS provider's control panel gives you power controls, OS reinstall and a browser console if the staging box gets stuck mid-boot.
  • SSH access to both hosts. Connect to your Linux VPS with SSH if this is your first login.
  • A maintenance window you can announce to whoever depends on the service.
  • A decision on which packages are critical: web server, database, PHP/Python runtime, kernel, and any vendor module (DKMS, monitoring agent).

Note: Repository contents change weekly. When a step here mentions a flag or subcommand, confirm it against your distribution's current documentation before you rely on it in a production window.

Build a Linux update strategy you can repeat

Work in three tiers and never skip one:

  1. Staging — a clone of production that receives every patch first.
  2. Canary — one low-traffic production host that receives the patch after staging passes.
  3. Fleet — the remaining production hosts, patched in small batches.

Three rules make the strategy hold together. Patch staging and production from the same repositories. Hold kernel and vendor-module updates until the new kernel has booted somewhere. Record the package versions you are replacing so you can put them back.

Step 1: Baseline what you run

Start by recording the current state so you can tell what changed.

Bash
cat /etc/os-release
uname -r

List pending updates. On Ubuntu and Debian:

Bash
sudo apt update
apt list --upgradable 2>/dev/null

On AlmaLinux and Rocky:

Bash
sudo dnf check-update
sudo dnf updateinfo list --security

Check whether anything already needs a restart:

Bash
sudo needrestart -b          # Ubuntu/Debian, install with: sudo apt install needrestart
sudo dnf needs-restarting -r # AlmaLinux/Rocky

Save a baseline file per host:

Bash
{ hostname; uname -r; cat /etc/os-release; } | sudo tee /root/baseline-$(date +%F).txt

Step 2: Stand up a staging server that matches production

Staging is only useful if it looks like production. Install the same distribution version, enable the same repositories, and install the same package set.

Export the package list from production:

Bash
dpkg --get-selections > /root/packages.list          # Ubuntu/Debian
rpm -qa --qf '%{NAME}\n' | sort > /root/packages.list # AlmaLinux/Rocky

Copy it to the staging host and rebuild, or restore the same configuration-management playbook you use for production. If you copy files directly, restrict yourself to what matters:

Bash
sudo rsync -aHAX --delete /etc/ deploy@203.0.113.10:/srv/staging/etc/

Warning: /etc can contain private keys, database passwords and mail credentials. Copy only the files you have reviewed, or move them through your secrets process rather than rsync.

Tip: A mismatched repo list is the single most common reason a patch passes on staging and breaks in production. Compare dnf repolist output or the files in /etc/apt/sources.list.d/ on both hosts before you start.

Step 3: Apply and test the patches on staging

Dry-run first so you can read the plan before anything is installed.

Bash
sudo apt-get -s upgrade
sudo apt-get -s dist-upgrade

On AlmaLinux and Rocky:

Bash
sudo dnf check-update --security
sudo dnf update --assumeno

Then apply. On Ubuntu and Debian:

Bash
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
  -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade

The two Dpkg::Options flags keep your existing configuration files instead of prompting. Review any .dpkg-dist files afterwards to see what the packager wanted to change.

On AlmaLinux and Rocky:

Bash
sudo dnf --security update

Restart if the kernel or a core library changed, then check the host:

Bash
sudo reboot

After it comes back:

Bash
uname -r
systemctl --failed
journalctl -p err -b --since "30 min ago"
ss -tulpn
dmesg --level=err,warn | tail -n 40
curl -fsS -o /dev/null -w '%{http_code}\n' https://example.com/health

Test the application the way a user would: log in, submit a form, run one query against the database. An HTTP 200 on a health endpoint does not prove the app works.

Warning: A new kernel can fail to boot if a third-party module (DKMS, storage driver, monitoring agent) has not been rebuilt for it. Keep console access through your VPS provider's control panel open during the first reboot after a kernel change.

Step 4: Roll out to a canary, then the rest

Once staging is healthy, freeze the exact versions you tested and move in small steps.

Apply the update to one production host first, run the same checks from Step 3, then wait through at least one full traffic cycle before continuing. Patch the remaining hosts in batches of two or three, checking systemctl --failed and application logs after each batch.

If updates arrive between windows and you need to keep a host pinned:

Bash
sudo apt-mark hold nginx          # Ubuntu/Debian
sudo dnf versionlock add nginx    # AlmaLinux/Rocky, from python3-dnf-plugin-versionlock

Remove the hold once the batch is finished, or the host will silently stop receiving security fixes.

Step 5: Keep a rollback path

Know how to reverse an update before you apply it.

On Ubuntu and Debian, the history log records every transaction:

Bash
grep -A3 "Start-Date" /var/log/apt/history.log | tail -n 30
sudo apt-get install nginx='<version-from-history>'

On AlmaLinux and Rocky, use transaction history:

Bash
sudo dnf history list
sudo dnf history undo <transaction-id>

For the kernel specifically, keep the previous kernel installed until the new one has booted successfully and your application has run for a full day. Third-party repositories do not keep old versions forever, so if a rollback depends on them, test it while the old package is still available.

Check it works

The rollout is done when all of these are true:

  • uname -r on every patched host returns the kernel version you tested on staging.
  • systemctl --failed returns no failed units.
  • journalctl -p err -b shows no new errors compared with your baseline.
  • sudo needrestart -b or sudo dnf needs-restarting -r reports no services still running old libraries.
  • df -h /boot has room for at least one more kernel, so the next update does not fail for lack of space.
  • Your application passes a real user-level test, not just a health check.

Troubleshooting

  • Could not get lock /var/lib/dpkg/lock-frontend — unattended-upgrades is running in the background. Wait for it to finish and check systemctl status unattended-upgrades. Never delete the lock file; you can leave the package database in an inconsistent state.
  • A service fails to start after the upgrade — the package shipped a new default config, or a shared library changed. Run journalctl -u <service> -n 50 and compare /etc/<service>/ against
Was this guide helpful?

Still stuck?

Ask MdmHost AI for a quick answer, or open a ticket and our team will take a look. We are here 24/7.

Ask MdmHost AIOpen a ticket

Related guides

Security

Set up a UFW firewall on Ubuntu and Debian

Set safe defaults, allow SSH before switching UFW on, open only the ports you use, limit sensitive ports to your own IP, and tidy up old rules.

3 min read
Security

Make iptables rules survive a reboot

iptables rules live in memory and disappear on reboot. Save them with netfilter-persistent on Debian and Ubuntu or iptables-services on AlmaLinux and Rocky Linux.

5 min read