Advanced Nginx load balancing with upstream directives

Nginx load balancing puts an upstream block in front of several application servers so one entry point spreads requests across them. This guide goes past the basic round-robin example and covers the directives that decide which backend gets a request, how failures are handled and how connections are reused.

You should already have Nginx serving a site and at least two backend servers or app instances reachable on private addresses. Every command below is run as root or with sudo on Ubuntu 22.04/24.04, Debian 12, AlmaLinux or Rocky Linux.

Before you start

• Nginx installed and running. If not, follow installing Nginx on Linux first.
• Two or more backends listening on a known address and port, for example 10.0.0.11:8080 and 10.0.0.12:8080. Placeholders in this guide: 203.0.113.10 for the public IP, example.com for the domain.
• The backends must answer healthily on their own before you add balancing. Test each one with curl -I http://10.0.0.11:8080/.
• A backup of the current config: cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak.
• Knowledge of where your site config lives. On Debian/Ubuntu it is /etc/nginx/sites-available/, on RHEL-family systems /etc/nginx/conf.d/.

Build the upstream block

The upstream block lives in the http context, alongside server blocks. Keep it in its own file so it is easy to find, for example /etc/nginx/conf.d/upstream-app.conf.

upstream app_backend {
    zone app_backend 64k;
    least_conn;

    server 10.0.0.11:8080 weight=3 max_fails=3 fail_timeout=10s;
    server 10.0.0.12:8080 weight=1 max_fails=3 fail_timeout=10s;
    server 10.0.0.13:8080 backup;

    keepalive 32;
    keepalive_requests 1000;
    keepalive_timeout 60s;
}

What each line does:

• zone reserves shared memory so workers see one consistent view of server state. It is required if you later add dynamic server management or use hash/random with zone.
• least_conn sends the next request to the backend with the fewest active connections, weighted by weight. Good when request times vary a lot.
• weight sets relative share. In the example the first server takes roughly three requests for every one sent to the second.
• max_fails and fail_timeout are the passive health check: after three failures within ten seconds, Nginx stops sending traffic there for ten seconds, then tries again.
• backup marks a server that only receives traffic when all primary servers are down.
• keepalive keeps idle connections to backends open, which removes a TCP and TLS handshake per request.

Then point your site at the upstream by name:

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_pass http://app_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 3s;
        proxy_read_timeout 60s;
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
    }
}

proxy_http_version 1.1 and the empty Connection header are both required for upstream keepalive. Without them Nginx talks HTTP/1.0 to the backend and closes every connection, and you will see the pool never being reused.

Choose the Nginx load balancing method

Nginx picks a method per upstream block. Only one method directive is allowed in each block.

• Round robin — the default, no directive needed. Requests go down the server list in order, adjusted by weights.
• least_conn — least connections first. Best for uneven response times.
• ip_hash — the client IP is hashed, so one visitor keeps hitting the same backend. Use it for apps that store sessions locally in memory or on disk.
• hash $key consistent — hash any variable, such as a cookie or a URL path. consistent keeps most keys on the same server when you add or remove one.
• random two least_conn — picks two servers at random and sends the request to the less busy of them. Useful with many backends.

A cookie-based sticky setup looks like this:

upstream app_backend {
    hash $cookie_sessionid consistent;
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
}

Handle failures and retries

The passive health check is only half of failure handling. proxy_next_upstream decides when Nginx gives up on one backend and tries the next.

• error — connection or send error to the backend.
• timeout — the connect, read or send timeout expired.
• http_502, http_503, http_504 — the backend answered with these codes.
• non_idempotent — allows retrying POST and other non-idempotent requests. It is not in the default set on purpose.

proxy_next_upstream_tries 2 caps the number of backends tried, so one request cannot walk the whole pool. Keep it small; each try costs latency.

For active health checks — probing backends on a schedule regardless of traffic — you need a commercial Nginx subscription or an external checker. Open source Nginx only has the passive method above. Check the vendor's documentation for the current feature split before planning around it.

If the backends use HTTPS, mark them accordingly:

upstream app_backend {
    server 10.0.0.11:8443;
    server 10.0.0.12:8443;
}

server {
    location / {
        proxy_pass https://app_backend;
        proxy_ssl_server_name on;
        proxy_ssl_name example.com;
    }
}

Check it works

Reload and confirm the syntax first:

sudo nginx -t
sudo systemctl reload nginx

Add a temporary debug header inside the location block so you can see which backend answered:

add_header X-Upstream $upstream_addr always;

Reload, then send a burst of requests:

for i in $(seq 1 12); do
  curl -sI http://203.0.113.10/ | grep -i x-upstream
done

You should see the addresses alternate roughly according to the weights. Then test failover: stop the app on 10.0.0.11 and repeat. Requests should land on 10.0.0.12 within fail_timeout seconds, not return 502 forever.

Check the error log while doing this:

sudo tail -f /var/log/nginx/error.log

Lines such as upstream server temporarily disabled confirm the passive check is working. Remove the debug header when you are finished.

Troubleshooting

• All requests hit the same backend. You are using ip_hash and testing from one IP, or a weight is far too high. Test from a second host, or switch to round robin to confirm the pool is healthy.
• 502 with Connection reset by peer on backends. Upstream keepalive is misconfigured: proxy_http_version 1.1 is missing or proxy_set_header Connection ""; was not added in the location block.
• 504 Gateway Time-out. The backend is slow, not down. Raise proxy_read_timeout and proxy_send_timeout, and check the app's own logs before blaming Nginx.
• Failover takes too long. fail_timeout also controls how long a failing server is skipped. Lower fail_timeout and max_fails for faster switching, but not so low that a single blip removes a healthy backend.
• duplicate writes after a retry. proxy_next_upstream retried a POST. Remove non_idempotent if you added it, and check proxy_next_upstream_tries.
• Config reload fails with unknown directive. The directive is only in a newer Nginx release or a commercial build. Run nginx -v and check the vendor's directive reference.

Next steps

• Start with the simpler walkthrough in load balance backends with Nginx upstream if you are new to reverse proxying.
• Add the proxy layer without an upstream block first: set up Nginx as a reverse proxy.
• Serving a websocket app? Read proxy WebSocket connections through Nginx before you change the upstream.
• Protect the front door with rate limiting in Nginx using limit_req.