Test firewall changes safely with an automatic rollback

Schedule an automatic undo before you change firewall rules. If a mistake blocks SSH, the old rules come back by themselves a few minutes later.

  • Intermediate
  • Linux VPS
  • 5 min read

By RunRacks Team · Published

Network engineers have an old habit: before you change a remote firewall, you schedule a job that undoes the change. If your new rules lock you out, the job restores access a few minutes later. If everything works, you cancel the job. This guide shows how to do that on a Linux VPS with UFW, nftables, firewalld and plain iptables.

Before you start

  • A RunRacks Linux VPS running Ubuntu 22.04/24.04, Debian 12, or AlmaLinux/Rocky Linux 8 or 9.
  • Root access or a user with sudo.
  • Your SSH port. Check it with sudo ss -tlnp | grep sshd.
  • The browser console in your Dashboard as a last resort. A rollback job makes it unlikely you will need it, but it is good to know it is there.

How the method works

  1. Save a copy of the current, working rules.
  2. Schedule a command that restores that copy in, say, five minutes.
  3. Apply the new rules.
  4. Open a new SSH session to test.
  5. If it works, cancel the scheduled restore. If you are locked out, wait. The restore runs on its own and you can log back in.

You can schedule the restore with systemd-run, which exists on every current systemd distribution, or with at, if it is installed.

Schedule the rollback with systemd-run

systemd-run creates a temporary timer that runs once. This example works for UFW:

Terminal
sudo cp -a /etc/ufw /root/ufw-backup
sudo systemd-run --on-active=5min --unit=fw-rollback \
  /bin/sh -c 'ufw --force disable'

Here the safe fallback is simply to turn UFW off. A firewall that is off is better than a server you cannot reach for the next five minutes.

Check the timer is waiting:

Terminal
systemctl list-timers fw-rollback.timer

Now make your change, for example:

Terminal
sudo ufw default deny incoming
sudo ufw allow 22/tcp
sudo ufw enable

Open a new SSH session from your computer. If it connects, cancel the rollback:

Terminal
sudo systemctl stop fw-rollback.timer

If you are locked out, wait five minutes and connect again. UFW will be off. Then fix the rule set from /root/ufw-backup and try again.

Schedule the rollback with at

at runs a command once at a set time. Install it if needed:

Terminal
sudo apt install at          # Ubuntu and Debian
sudo dnf install at          # AlmaLinux and Rocky Linux
sudo systemctl enable --now atd

Queue the rollback. This example restores a saved nftables ruleset:

Terminal
sudo nft list ruleset > /root/nft-good.conf
echo 'nft flush ruleset && nft -f /root/nft-good.conf' | sudo at now + 5 minutes

at prints a job number. List and cancel jobs like this:

Terminal
sudo atq
sudo atrm 3

Replace 3 with your job number.

Use iptables-apply for iptables rules

On Debian and Ubuntu, the iptables package includes iptables-apply, which builds the rollback in. It loads a rules file, then asks you to confirm within a timeout. If you do not answer, because your connection dropped, it restores the previous rules.

Terminal
sudo iptables-save > /root/iptables-good.rules
sudo nano /etc/iptables/rules.v4
sudo iptables-apply -t 60 /etc/iptables/rules.v4

When it asks Can you establish NEW connections to the machine? (y/N), open a second SSH session first. Answer y only if that works. If the session hangs, do nothing and wait 60 seconds.

firewalld: runtime first, permanent later

firewalld has a built-in safety pattern. Changes without --permanent only affect the running firewall, so a reload or restart undoes them. Combine that with a timer:

Terminal
sudo systemd-run --on-active=5min --unit=fw-rollback \
  firewall-cmd --reload
sudo firewall-cmd --remove-service=ssh   # example of a risky runtime change

The scheduled --reload throws away runtime changes and restores the saved permanent config. When you are happy with the runtime state, make it permanent and cancel the timer:

Terminal
sudo firewall-cmd --runtime-to-permanent
sudo systemctl stop fw-rollback.timer

Check it works

Before you rely on this method, test it once on purpose:

  1. Schedule a rollback for two minutes.
  2. Apply a rule that blocks your SSH port, for example sudo ufw deny 22/tcp with UFW enabled.
  3. Watch your session freeze.
  4. Wait two minutes and reconnect. You should get in.

You now know the safety net works on this server.

Troubleshooting

  • Rollback did not run: check journalctl -u fw-rollback or sudo atq. With at, make sure atd is running.
  • Rollback ran but SSH is still blocked: the backup itself was bad, or the restore command failed. Log in via the browser console and run the restore by hand.
  • You forgot to cancel and the rollback undid good rules: apply the new rules again and cancel the job this time.
  • Existing SSH session stayed open, so you thought it was fine: established connections often survive a rule change. Always test with a new session.
  • `systemd-run` says the unit already exists: pick a different --unit name or reset the old one.

Next steps

Was this guide helpful?

Still stuck?

Ask RunRacks AI for a quick answer, or open a ticket and our team will take a look at your server.

Security

Check the Windows Firewall rule for RDP port 12230

Find and verify the inbound TCP 12230 rule that lets Remote Desktop reach your RunRacks server, recreate it if it was deleted and scope it without locking yourself out.

  • Intermediate
  • Windows VPS · Admin RDP
  • 5 min read

Security

iptables basics: list, allow, drop and save rules

Learn how iptables chains and rule order work, then build a small ruleset that allows SSH and web traffic, drops everything else and can be edited by line number.

  • Intermediate
  • Linux VPS
  • 6 min read