Linux Firewall: Using iptables to Implement Connection Tracking Limits

Cap new and simultaneous connections on a Linux VPS with iptables conntrack, limit, connlimit and hashlimit rules, and size the conntrack table so it does not fill up.

  • Advanced
  • Linux VPS
  • 7 min read

By RunRacks Team · Published

iptables connection tracking limits let you cap how many new and how many simultaneous connections a host or a single IP can open, which stops SYN floods and connection-table exhaustion from taking a small VPS offline. This guide walks through the conntrack, limit, connlimit and hashlimit rules that do the work, then shows how to size the tracking table so it does not fill up under load.

Before you start

You need:

  • A Linux VPS with root or a sudo user. Ubuntu 22.04/24.04, Debian 12, AlmaLinux 8–9 and Rocky 8–9 all work the same way here.
  • iptables installed. Most current distros ship iptables-nft, which accepts the same syntax and is fine for this guide.
  • The conntrack tool for inspecting the tracking table.

Install the packages:

Terminal
sudo apt update && sudo apt install -y iptables conntrack
Terminal
sudo dnf install -y iptables iptables-services conntrack-tools

Check which backend you are on:

Terminal
iptables --version

If the output mentions nf_tables, your rules are translated to nftables underneath. The commands below still work, but read UFW vs firewalld vs nftables vs iptables if you would rather manage the same policy with a different tool.

Check whether connection tracking is active

Connection tracking is on by default whenever any rule uses -m conntrack or -m state. Confirm the table exists and see how full it is:

Terminal
sudo sysctl net.netfilter.nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_count

The first value is the maximum number of tracked flows, the second is how many are in use right now. For a live view:

Terminal
sudo conntrack -C
sudo conntrack -S

conntrack -C prints the current entry count, conntrack -S prints per-CPU statistics including insert_failed and drop, which rise when the table is full. If nf_conntrack_max does not exist, load the module:

Terminal
sudo modprobe nf_conntrack

Apply iptables connection tracking limits

Build the rules in order. Each block assumes the previous one is already in place.

Accept established traffic first

Terminal
sudo iptables -A INPUT -i lo -j ACCEPT
sudo iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

The INVALID drop matters: malformed packets that match no known flow are a common part of scanning and flood traffic, and they cost tracking entries while they sit around.

Rate-limit new connections per source

The limit module caps how often a rule can match. It is global by default, not per IP, so it suits a single service such as SSH:

Terminal
sudo iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m limit --limit 4/minute --limit-burst 6 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j DROP

The first rule accepts the first six new SSH connections in a burst, then four per minute. Everything past that hits the second rule and is dropped. Adjust the numbers to match how your team actually connects.

Cap simultaneous connections per IP with connlimit

connlimit counts concurrent connections from one source address and rejects the ones above your threshold:

Terminal
sudo iptables -A INPUT -p tcp --dport 80 -m connlimit --connlimit-above 50 --connlimit-mask 32 -j REJECT --reject-with tcp-reset

--connlimit-mask 32 means one IPv4 address. Use 24 if you want to treat a whole /24 as a single client, which is useful when many users sit behind one NAT gateway. Option names changed between iptables releases, so verify yours:

Terminal
iptables -m connlimit --help

Per-IP rate limiting with hashlimit

hashlimit keeps a separate counter per source address, which is what you want on web ports where a global limit would punish everyone:

Terminal
sudo iptables -A INPUT -p tcp --dport 443 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name https_new --hashlimit-above 30/sec --hashlimit-mode srcip --hashlimit-burst 60 -j DROP

Each IP gets its own 30-per-second budget with a burst of 60. The --hashlimit-name value creates a table in /proc/net/ipt_hashlimit/ that you can read to see who is being throttled.

Tune the conntrack table

Limits only help if the table itself has room. Create a sysctl drop-in:

INI
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_established = 3600
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
net.netfilter.nf_conntrack_udp_timeout = 30

Save it as /etc/sysctl.d/99-conntrack.conf and apply:

Terminal
sudo sysctl --system

Each tracked entry costs roughly a few hundred bytes of kernel memory, so size nf_conntrack_max against your VPS RAM rather than picking the largest number you can find. Shortening the established and TIME_WAIT timeouts frees entries faster on a busy web server.

The hash bucket count, net.netfilter.nf_conntrack_buckets, is read-only on many current kernels. Set it as a module option instead:

INI
options nf_conntrack hashsize=65536

Put that in /etc/modprobe.d/nf_conntrack.conf. It takes effect after the module reloads, which normally means a reboot.

Make the rules persistent

Rules vanish on reboot unless you save them.

On Ubuntu and Debian:

Terminal
sudo apt install -y iptables-persistent
sudo netfilter-persistent save

On AlmaLinux and Rocky:

Terminal
sudo dnf install -y iptables-services
sudo systemctl enable --now iptables
sudo iptables-save | sudo tee /etc/sysconfig/iptables

Re-run the save command after every change, or your next reboot silently reverts to the old policy.

Check it works

List the rules with packet counters so you can see which ones are matching:

Terminal
sudo iptables -L INPUT -v -n --line-numbers

Then watch the tracking table while you generate load from another machine:

Terminal
sudo conntrack -L | wc -l
sudo conntrack -S
ss -s

Counters on the limit, connlimit and hashlimit rules should climb when you hammer the port with a loop of curl or nc requests, and insert_failed in conntrack -S should stay at zero. If you want to break down what is actually connected, netstat and ss for connection analysis covers the socket side in detail.

Troubleshooting

  • `nf_conntrack: table full, dropping packet` in `dmesg` — the table hit nf_conntrack_max. Raise the value, shorten the established and TIME_WAIT timeouts, or reduce the number of long-lived connections your services keep open.
  • Your SSH session freezes the moment you apply the rules — the ESTABLISHED,RELATED accept rule is missing or sits below a DROP. Recover through the browser console in the Dashboard, then reorder the chain so established traffic is accepted first.
  • `connlimit: invalid option --connlimit-above` — your iptables build uses older option names. Run iptables -m connlimit --help and use the spelling it lists.
  • Legitimate users behind one office NAT get rejected — --connlimit-mask 32 treats each public IP as one client. Raise the mask to 24 or increase the threshold.
  • All rules are gone after a reboot — you did not persist them. Run netfilter-persistent save or write /etc/sysconfig/iptables as shown above.

Next steps

Was this guide helpful?

Still stuck?

Ask RunRacks AI for a quick answer, or open a ticket and our team will take a look at your server.

Security

Check the Windows Firewall rule for RDP port 12230

Find and verify the inbound TCP 12230 rule that lets Remote Desktop reach your RunRacks server, recreate it if it was deleted and scope it without locking yourself out.

  • Intermediate
  • Windows VPS · Admin RDP
  • 5 min read

Security

iptables basics: list, allow, drop and save rules

Learn how iptables chains and rule order work, then build a small ruleset that allows SSH and web traffic, drops everything else and can be edited by line number.

  • Intermediate
  • Linux VPS
  • 6 min read