Linux Firewall: Using iptables to Implement Connection Tracking Limits

iptables connection tracking limits let you cap how many new and how many simultaneous connections a host or a single IP can open, which stops SYN floods and connection-table exhaustion from taking a small VPS offline. This guide walks through the conntrack, limit, connlimit and hashlimit rules that do the work, then shows how to size the tracking table so it does not fill up under load.

Before you start

You need:

• A Linux VPS with root or a sudo user. Ubuntu 22.04/24.04, Debian 12, AlmaLinux 8–9 and Rocky 8–9 all work the same way here.
• iptables installed. Most current distros ship iptables-nft, which accepts the same syntax and is fine for this guide.
• The conntrack tool for inspecting the tracking table.

Install the packages:

sudo apt update && sudo apt install -y iptables conntrack

sudo dnf install -y iptables iptables-services conntrack-tools

Check which backend you are on:

iptables --version

If the output mentions nf_tables, your rules are translated to nftables underneath. The commands below still work, but read UFW vs firewalld vs nftables vs iptables if you would rather manage the same policy with a different tool.

Check whether connection tracking is active

Connection tracking is on by default whenever any rule uses -m conntrack or -m state. Confirm the table exists and see how full it is:

sudo sysctl net.netfilter.nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_count

The first value is the maximum number of tracked flows, the second is how many are in use right now. For a live view:

sudo conntrack -C
sudo conntrack -S

conntrack -C prints the current entry count, conntrack -S prints per-CPU statistics including insert_failed and drop, which rise when the table is full. If nf_conntrack_max does not exist, load the module:

sudo modprobe nf_conntrack

Apply iptables connection tracking limits

Build the rules in order. Each block assumes the previous one is already in place.

Accept established traffic first

sudo iptables -A INPUT -i lo -j ACCEPT
sudo iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

The INVALID drop matters: malformed packets that match no known flow are a common part of scanning and flood traffic, and they cost tracking entries while they sit around.

Rate-limit new connections per source

The limit module caps how often a rule can match. It is global by default, not per IP, so it suits a single service such as SSH:

sudo iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m limit --limit 4/minute --limit-burst 6 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j DROP

The first rule accepts the first six new SSH connections in a burst, then four per minute. Everything past that hits the second rule and is dropped. Adjust the numbers to match how your team actually connects.

Cap simultaneous connections per IP with connlimit

connlimit counts concurrent connections from one source address and rejects the ones above your threshold:

sudo iptables -A INPUT -p tcp --dport 80 -m connlimit --connlimit-above 50 --connlimit-mask 32 -j REJECT --reject-with tcp-reset

--connlimit-mask 32 means one IPv4 address. Use 24 if you want to treat a whole /24 as a single client, which is useful when many users sit behind one NAT gateway. Option names changed between iptables releases, so verify yours:

iptables -m connlimit --help

Per-IP rate limiting with hashlimit

hashlimit keeps a separate counter per source address, which is what you want on web ports where a global limit would punish everyone:

sudo iptables -A INPUT -p tcp --dport 443 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name https_new --hashlimit-above 30/sec --hashlimit-mode srcip --hashlimit-burst 60 -j DROP

Each IP gets its own 30-per-second budget with a burst of 60. The --hashlimit-name value creates a table in /proc/net/ipt_hashlimit/ that you can read to see who is being throttled.

Tune the conntrack table

Limits only help if the table itself has room. Create a sysctl drop-in:

net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_established = 3600
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
net.netfilter.nf_conntrack_udp_timeout = 30

Save it as /etc/sysctl.d/99-conntrack.conf and apply:

sudo sysctl --system

Each tracked entry costs roughly a few hundred bytes of kernel memory, so size nf_conntrack_max against your VPS RAM rather than picking the largest number you can find. Shortening the established and TIME_WAIT timeouts frees entries faster on a busy web server.

The hash bucket count, net.netfilter.nf_conntrack_buckets, is read-only on many current kernels. Set it as a module option instead:

options nf_conntrack hashsize=65536

Put that in /etc/modprobe.d/nf_conntrack.conf. It takes effect after the module reloads, which normally means a reboot.

Make the rules persistent

Rules vanish on reboot unless you save them.

On Ubuntu and Debian:

sudo apt install -y iptables-persistent
sudo netfilter-persistent save

On AlmaLinux and Rocky:

sudo dnf install -y iptables-services
sudo systemctl enable --now iptables
sudo iptables-save | sudo tee /etc/sysconfig/iptables

Re-run the save command after every change, or your next reboot silently reverts to the old policy.

Check it works

List the rules with packet counters so you can see which ones are matching:

sudo iptables -L INPUT -v -n --line-numbers

Then watch the tracking table while you generate load from another machine:

sudo conntrack -L | wc -l
sudo conntrack -S
ss -s

Counters on the limit, connlimit and hashlimit rules should climb when you hammer the port with a loop of curl or nc requests, and insert_failed in conntrack -S should stay at zero. If you want to break down what is actually connected, netstat and ss for connection analysis covers the socket side in detail.

Troubleshooting

• nf_conntrack: table full, dropping packet in dmesg — the table hit nf_conntrack_max. Raise the value, shorten the established and TIME_WAIT timeouts, or reduce the number of long-lived connections your services keep open.
• Your SSH session freezes the moment you apply the rules — the ESTABLISHED,RELATED accept rule is missing or sits below a DROP. Recover through the browser console in your VPS provider's control panel, then reorder the chain so established traffic is accepted first.
• connlimit: invalid option --connlimit-above — your iptables build uses older option names. Run iptables -m connlimit --help and use the spelling it lists.
• Legitimate users behind one office NAT get rejected — --connlimit-mask 32 treats each public IP as one client. Raise the mask to 24 or increase the threshold.
• All rules are gone after a reboot — you did not persist them. Run netfilter-persistent save or write /etc/sysconfig/iptables as shown above.

Next steps

• Using ipset for high-performance IP blacklisting to block large address lists without bloating the rule set.
• Using netstat and ss for detailed network connection analysis to see which processes own the connections you are limiting.
• Linux incident response: isolate a compromised server if the flood turns out to be part of a breach rather than noise.