Linux Firewall: Using nftables to Drop Malformed Packets and Attack Vectors

To drop malformed packets with nftables you inspect TCP flag combinations, conntrack state and ICMP types in a prerouting chain that runs before your normal filter rules, then drop anything that could not have come from a legitimate TCP/IP stack. This guide builds that chain step by step on a Linux VPS, adds rate limits for the noisiest attack vectors, and shows you how to confirm the counters are moving.

The rules below work on Ubuntu 22.04/24.04, Debian 12 and AlmaLinux/Rocky 8–9 with the kernel nftables stack. Nothing here replaces a stateful filter policy — it sits in front of it.

Before you start

• Root or sudo access on the VPS, and a second SSH session or console open in your VPS provider's control panel in case a rule locks you out.
• nft installed. On Debian and Ubuntu: sudo apt install nftables. On AlmaLinux and Rocky: sudo dnf install nftables.
• A baseline ruleset you can restore. Save it first:

sudo nft list ruleset | sudo tee /root/nftables-backup-$(date +%F).rules

• Know which ports must stay open (SSH, HTTP/HTTPS, mail). If you are still choosing a firewall front end, read UFW vs firewalld vs nftables vs iptables first.

Understand the hook order

nftables runs chains in priority order on each hook. Conntrack attaches at priority -200, so a chain at priority raw (-300) sees packets before the connection tracker does. That matters: ct state invalid is meaningless in a raw chain because no state exists yet.

Use two chains:

• raw_prerouting at priority raw — TCP flag sanity checks, fragment checks, obviously bogus headers.
• input at priority filter — ct state invalid, rate limits, and your normal accept/drop policy.

Create the malformed packet chain

Create the table and the raw chain:

sudo nft add table inet filter
sudo nft add chain inet filter raw_prerouting '{ type filter hook prerouting priority raw; policy accept; }'

Now drop TCP packets whose flags cannot occur in real traffic. A NULL scan sends no flags at all; an XMAS scan sets FIN, PSH and URG together; SYN+FIN and SYN+RST are impossible combinations.

sudo nft add rule inet filter raw_prerouting tcp flags & (fin|syn|rst|psh|ack|urg) == 0 counter drop
sudo nft add rule inet filter raw_prerouting tcp flags & (fin|syn) == (fin|syn) counter drop
sudo nft add rule inet filter raw_prerouting tcp flags & (syn|rst) == (syn|rst) counter drop
sudo nft add rule inet filter raw_prerouting tcp flags & (fin|rst) == (fin|rst) counter drop
sudo nft add rule inet filter raw_prerouting tcp flags & (fin|psh|urg) == (fin|psh|urg) counter drop

Drop TCP with a zero source or destination port, and TCP with a zero window on a SYN (a common evasion trick):

sudo nft add rule inet filter raw_prerouting tcp sport 0 counter drop
sudo nft add rule inet filter raw_prerouting tcp dport 0 counter drop
sudo nft add rule inet filter raw_prerouting tcp flags syn tcp window 0 counter drop

Fragmented traffic is mostly noise on a web or application server. If you do not run a service that needs fragments, drop the non-first fragments:

sudo nft add rule inet filter raw_prerouting ip frag-off & 0x1fff != 0 counter drop

Drop invalid conntrack state and rate-limit scans

Create the input chain and drop packets the connection tracker has already marked invalid:

sudo nft add chain inet filter input '{ type filter hook input priority filter; policy accept; }'
sudo nft add rule inet filter input ct state invalid counter drop
sudo nft add rule inet filter input iif lo counter accept
sudo nft add rule inet filter input ct state established,related counter accept

Reject new TCP connections that do not start with a bare SYN. This kills ACK scans and FIN scans that survived the raw chain:

sudo nft add rule inet filter input tcp flags & (fin|syn|rst|ack) != syn ct state new counter drop

Rate-limit ICMP echo requests so a ping flood cannot saturate the link, and drop the obsolete ICMP types that should never arrive:

sudo nft add rule inet filter input icmp type echo-request limit rate 10/second burst 20 packets counter accept
sudo nft add rule inet filter input icmp type echo-request counter drop
sudo nft add rule inet filter input icmp type { redirect, router-advertisement, router-solicitation } counter drop

Add a per-source meter on SSH so brute-force attempts are throttled rather than blocked outright:

sudo nft add rule inet filter input tcp dport 22 ct state new meter sshflood '{ ip saddr limit rate 4/minute burst 6 packets }' counter accept
sudo nft add rule inet filter input tcp dport 22 ct state new counter drop

Then allow the services you actually run and drop the rest:

sudo nft add rule inet filter input tcp dport { 80, 443 } counter accept
sudo nft add rule inet filter input counter drop

Persist everything so it survives a reboot:

sudo nft list ruleset | sudo tee /etc/nftables.conf
sudo systemctl enable --now nftables

Check it works

List the ruleset with counters and handle numbers:

sudo nft -a list ruleset

The counter values show how many packets each rule matched. After a few minutes of normal traffic, the malformed-packet rules should sit at or near zero — if one climbs fast, something upstream is sending broken traffic and you should investigate rather than assume an attack.

Test from a second host. A NULL scan should be dropped and counted:

sudo nmap -sN -p 22,80,443 203.0.113.10

Watch the counters while you scan:

watch -n 1 'sudo nft list chain inet filter raw_prerouting'

Confirm your own access still works before you close the console:

ssh deploy@203.0.113.10
curl -I https://example.com

Troubleshooting

• SSH session freezes after applying rules — the input policy or a drop rule caught your connection. Reconnect through your VPS provider's console, run sudo nft flush ruleset, then reload your backup with sudo nft -f /root/nftables-backup-*.rules.
• ct state invalid never matches in the raw chain — conntrack runs at priority -200, after raw. Move that rule to the input chain at priority filter, as shown above.
• Legitimate clients get dropped as malformed — some load balancers and NAT devices rewrite flags. Check the counters per rule with nft -a list ruleset, then narrow the rule to the offending interface with iif instead of dropping globally.
• Fragmented VPN or tunnel traffic breaks — remove the ip frag-off rule. Only keep it if you have confirmed nothing you run depends on fragments.
• Rules vanish after reboot — the ruleset was never written to /etc/nftables.conf, or the nftables service is disabled. Run sudo systemctl enable --now nftables and check systemctl status nftables.

Next steps

• Layer stateful policy on top of these drops with nftables complex rule chaining for stateful filtering.
• Add per-IP connection limits for services that open many sockets, covered in iptables connection tracking limits.
• Harden remote access further with disabling weak SSH ciphers and MAC algorithms.
• If you have not finished the base build yet, work through first steps after your Linux VPS is ready.