Linux Update Strategy: Testing Security Patches Before Production

Build a Linux update strategy that tests security patches on a staging server first, then promotes them to production with a canary rollout and a rollback path.

  • Advanced
  • Linux VPS
  • 6 min read

By RunRacks Team · Published

A Linux update strategy that tests security patches before production keeps one bad kernel or library from taking down every server at once. This guide builds a staging tier, applies security and kernel updates there first, and promotes them to production only after your checks pass.

Before you start

You need:

  • A Linux VPS running Ubuntu 22.04/24.04, Debian 12, AlmaLinux 8–9 or Rocky 8–9, with root or a sudo user — see creating a sudo user if you haven't yet.
  • A second VPS to act as staging. The Dashboard at /dashboard gives you power controls, OS reinstall and a browser console if the staging box gets stuck mid-boot.
  • SSH access to both hosts. Connect to your Linux VPS with SSH if this is your first login.
  • A maintenance window you can announce to whoever depends on the service.
  • A decision on which packages are critical: web server, database, PHP/Python runtime, kernel, and any vendor module (DKMS, monitoring agent).

Build a Linux update strategy you can repeat

Work in three tiers and never skip one:

  1. Staging — a clone of production that receives every patch first.
  2. Canary — one low-traffic production host that receives the patch after staging passes.
  3. Fleet — the remaining production hosts, patched in small batches.

Three rules make the strategy hold together. Patch staging and production from the same repositories. Hold kernel and vendor-module updates until the new kernel has booted somewhere. Record the package versions you are replacing so you can put them back.

Step 1: Baseline what you run

Start by recording the current state so you can tell what changed.

Terminal
cat /etc/os-release
uname -r

List pending updates. On Ubuntu and Debian:

Terminal
sudo apt update
apt list --upgradable 2>/dev/null

On AlmaLinux and Rocky:

Terminal
sudo dnf check-update
sudo dnf updateinfo list --security

Check whether anything already needs a restart:

Terminal
sudo needrestart -b          # Ubuntu/Debian, install with: sudo apt install needrestart
sudo dnf needs-restarting -r # AlmaLinux/Rocky

Save a baseline file per host:

Terminal
{ hostname; uname -r; cat /etc/os-release; } | sudo tee /root/baseline-$(date +%F).txt

Step 2: Stand up a staging server that matches production

Staging is only useful if it looks like production. Install the same distribution version, enable the same repositories, and install the same package set.

Export the package list from production:

Terminal
dpkg --get-selections > /root/packages.list          # Ubuntu/Debian
rpm -qa --qf '%{NAME}\n' | sort > /root/packages.list # AlmaLinux/Rocky

Copy it to the staging host and rebuild, or restore the same configuration-management playbook you use for production. If you copy files directly, restrict yourself to what matters:

Terminal
sudo rsync -aHAX --delete /etc/ [email protected]:/srv/staging/etc/

Step 3: Apply and test the patches on staging

Dry-run first so you can read the plan before anything is installed.

Terminal
sudo apt-get -s upgrade
sudo apt-get -s dist-upgrade

On AlmaLinux and Rocky:

Terminal
sudo dnf check-update --security
sudo dnf update --assumeno

Then apply. On Ubuntu and Debian:

Terminal
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
  -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade

The two Dpkg::Options flags keep your existing configuration files instead of prompting. Review any .dpkg-dist files afterwards to see what the packager wanted to change.

On AlmaLinux and Rocky:

Terminal
sudo dnf --security update

Restart if the kernel or a core library changed, then check the host:

Terminal
sudo reboot

After it comes back:

Terminal
uname -r
systemctl --failed
journalctl -p err -b --since "30 min ago"
ss -tulpn
dmesg --level=err,warn | tail -n 40
curl -fsS -o /dev/null -w '%{http_code}\n' https://example.com/health

Test the application the way a user would: log in, submit a form, run one query against the database. An HTTP 200 on a health endpoint does not prove the app works.

Step 4: Roll out to a canary, then the rest

Once staging is healthy, freeze the exact versions you tested and move in small steps.

Apply the update to one production host first, run the same checks from Step 3, then wait through at least one full traffic cycle before continuing. Patch the remaining hosts in batches of two or three, checking systemctl --failed and application logs after each batch.

If updates arrive between windows and you need to keep a host pinned:

Terminal
sudo apt-mark hold nginx          # Ubuntu/Debian
sudo dnf versionlock add nginx    # AlmaLinux/Rocky, from python3-dnf-plugin-versionlock

Remove the hold once the batch is finished, or the host will silently stop receiving security fixes.

Step 5: Keep a rollback path

Know how to reverse an update before you apply it.

On Ubuntu and Debian, the history log records every transaction:

Terminal
grep -A3 "Start-Date" /var/log/apt/history.log | tail -n 30
sudo apt-get install nginx='<version-from-history>'

On AlmaLinux and Rocky, use transaction history:

Terminal
sudo dnf history list
sudo dnf history undo <transaction-id>

For the kernel specifically, keep the previous kernel installed until the new one has booted successfully and your application has run for a full day. Third-party repositories do not keep old versions forever, so if a rollback depends on them, test it while the old package is still available.

Check it works

The rollout is done when all of these are true:

  • uname -r on every patched host returns the kernel version you tested on staging.
  • systemctl --failed returns no failed units.
  • journalctl -p err -b shows no new errors compared with your baseline.
  • sudo needrestart -b or sudo dnf needs-restarting -r reports no services still running old libraries.
  • df -h /boot has room for at least one more kernel, so the next update does not fail for lack of space.
  • Your application passes a real user-level test, not just a health check.

Troubleshooting

  • `Could not get lock /var/lib/dpkg/lock-frontend` — unattended-upgrades is running in the background. Wait for it to finish and check systemctl status unattended-upgrades. Never delete the lock file; you can leave the package database in an inconsistent state.
  • A service fails to start after the upgrade — the package shipped a new default config, or a shared library changed. Run journalctl -u <service> -n 50 and compare /etc/<service>/ against

Was this guide helpful?

Still stuck?

Ask RunRacks AI for a quick answer, or open a ticket and our team will take a look at your server.

Security

Check the Windows Firewall rule for RDP port 12230

Find and verify the inbound TCP 12230 rule that lets Remote Desktop reach your RunRacks server, recreate it if it was deleted and scope it without locking yourself out.

  • Intermediate
  • Windows VPS · Admin RDP
  • 5 min read

Security

iptables basics: list, allow, drop and save rules

Learn how iptables chains and rule order work, then build a small ruleset that allows SSH and web traffic, drops everything else and can be edited by line number.

  • Intermediate
  • Linux VPS
  • 6 min read