Auto-scaling Redis broker: with and without broadcast

A horizontally scalable Redis broker is the missing half of scaling a message-driven system. Adding service instances is easy — with @imqueue two copies of a service just read the same queue — but all of that traffic still funnels through one Redis. At some point that single broker is both your throughput ceiling and your single point of failure. @imqueue's answer is not Redis Cluster and not a managed proxy: it's a fleet of plain, independent Redis instances that services discover at runtime, with producers spreading load across them and consumers draining all of them at once. And because discovery runs continuously, the fleet doesn't just scale — it auto-scales: add a broker and every service folds it into rotation within a second; remove one and traffic re-routes just as fast, no config pushes, no redeploys. The only part that changes between environments is how brokers announce themselves — and that's what the two recipes below are about.

TL;DR — Load a tiny announcer module into every Redis broker and the broker layer becomes horizontally auto-scaling: services discover the fleet over UDP as brokers come and go. On networks that deliver limited broadcast (bare metal, LANs, Docker bridge) use redis-broker-promoter, which shouts to 255.255.255.255. On networks that drop broadcast — GCP VPCs, most Kubernetes overlays — use redis-broker-unicaster, which asks the Kubernetes API for pod IPs and unicasts the same datagram to each of them. The service side is identical either way: clusterManagers: [new UDPClusterManager()].

How @imqueue clusters the broker

Clustering lives on the client side, in ClusteredRedisQueue. You never instantiate it directly — the factory swaps it in whenever the options mention a cluster, so services and generated clients get it with zero code changes:

import { IMQServiceOptions, UDPClusterManager } from '@imqueue/rpc';

export const serviceOptions: Partial<IMQServiceOptions> = {
    // dynamic discovery (the subject of this post):
    clusterManagers: [new UDPClusterManager()],
    // …or, instead of a manager, a static fleet known up front:
    // cluster: [
    //     { host: 'redis-1', port: 6379 },
    //     { host: 'redis-2', port: 6379 },
    // ],
};

The model is deliberately simple. There is no sharding and no consistent hashing: every broker hosts an identically-named queue, producers pick a broker per message in health-aware round-robin (a broker whose connection is known to be down is skipped), and consumers run a blocking read against all brokers concurrently. Throughput scales with the number of brokers; losing one broker just narrows the rotation. The brokers themselves are stock standalone Redis — they never talk to each other, don't replicate, and don't even know they are part of a fleet. Neither announcer module registers a single Redis command.

The discovery protocol both recipes share

Each broker loads a small C module that periodically emits a one-line, tab-separated UDP datagram:

imq-broker  2cc7c345-3569-44bb-b57a-b72d729d7012  up    10.0.4.12:6379  1  plain
imq-broker  2cc7c345-3569-44bb-b57a-b72d729d7012  down  10.0.4.12:6379

That's name, a per-process GUID, up/down, the advertised host:port, and — for up — the announce interval in seconds and whether that port speaks TLS (tls or plain, from redis-broker v1.2.0; see encrypting the fleet). On the service side, UDPClusterManager listens on UDP port 63000 (its default) in a worker thread and translates datagrams into cluster changes:

• up — add the broker (deduplicated by GUID or address) and re-arm its liveness timer.
• down — sent on graceful shutdown; the broker is removed immediately.
• silence — a broker that misses heartbeats for interval × 1000 + 5000 + 1 ms (about six seconds at the default 1-second interval) is evicted, which covers crashes and network partitions.

Both modules read the same environment variables — REDIS_BROADCAST_NAME (default imq-broker), REDIS_BROADCAST_INTERVAL (seconds, default 1), REDIS_BROADCAST_TLS (unset, and covered under encrypting the fleet) — and emit byte-identical messages. They differ only in how the datagram travels, which is exactly why the client side doesn't care which one you run.

Recipe 1: networks that deliver broadcast — redis-broker-promoter

If your brokers and services share an L2 segment — bare-metal boxes, on-prem VMs, a Docker bridge network, your laptop — the simplest transport is UDP limited broadcast: one sendto() to 255.255.255.255 reaches every host on the segment, no inventory required. That's all redis-broker-promoter does:

docker run -p 6379:6379 \
    -e IMQ_BROKER_MODE=promoter \
    -e REDIS_BROADCAST_NAME=imq-broker \
    -e REDIS_BROADCAST_INTERVAL=1 \
    ghcr.io/imqueue/redis-broker:7.4

That image is imqueue/redis-broker — Redis with both announcer modules built in, one of them selected at runtime by IMQ_BROKER_MODE. It is the same module either way; if you would rather build it yourself, the source is two files and a Makefile:

git clone https://github.com/imqueue/redis-broker-promoter.git
cd redis-broker-promoter && make   # needs libuuid
redis-server --port 6379 --loadmodule $PWD/promoter.so

On load the module spawns one announcer thread per network interface allowed by your Redis bind configuration (0.0.0.0 means all of them) and broadcasts up every interval to 255.255.255.255:63000 (REDIS_BROADCAST_PORT configurable). On shutdown it broadcasts down. Scaling out is now an operational no-op: start another redis-server with the module loaded, and every service adds it to the rotation within roughly one interval. Stop it, and the fleet shrinks just as automatically. That is horizontal auto-scaling of the broker layer — hook broker instances to whatever triggers your scaling decisions and the services follow along; nothing else needs restarting or reconfiguring.

A useful side effect of limited broadcast: routers never forward 255.255.255.255, so announcements are confined to the local segment. That's the constraint that breaks this recipe in the cloud — and a small security property everywhere else.

Recipe 2: networks that block broadcast — redis-broker-unicaster (Kubernetes on GCP and other clouds)

Cloud VPCs are software-defined networks, and most of them — GCP explicitly — do not deliver broadcast or multicast at all. A datagram to 255.255.255.255 in a GCP VPC or across a typical Kubernetes overlay simply vanishes, and the promoter recipe goes silent.

redis-broker-unicaster emulates broadcast instead of relying on it. Every interval it asks the Kubernetes API for the pods in its namespace and sends the very same datagram as plain UDP unicast to each pod IP at port 63000. Pods that aren't listening drop it; pods running UDPClusterManager get exactly what they would have gotten from a broadcast. Newly scheduled service pods start receiving announcements within one interval — no service registry, no headless-service DNS, no multicast anywhere. Scale the broker Deployment up or down — by hand or with an autoscaler — and the fleet follows: the same horizontal auto-scaling as the broadcast recipe, minus the broadcast.

One boundary to be clear about: this recipe lives inside Kubernetes — the module authenticates with the pod's mounted service-account token and talks to kubernetes.default.svc. On cloud VMs outside Kubernetes, reach for the static cluster list instead (last row of the table below). The broker's service account needs permission to list pods:

# in the broker pod spec — the module needs the mounted service-account token
containers:
  - name: redis
    image: ghcr.io/imqueue/redis-broker:7.4
    env:
      - { name: IMQ_BROKER_MODE, value: unicaster }
      - name: DEPLOYMENT_ENV          # the NAMESPACE — see below
        valueFrom:
          fieldRef: { fieldPath: metadata.namespace }
      - { name: SELECTED_INTERFACES, value: "10." }

Only the mode changes between the two recipes, which is why one image carries both modules — you often cannot answer "does this network deliver broadcast?" until the pod is scheduled. deploy/unicaster/ in that repo has the ServiceAccount, the Role and the NetworkPolicy to go with it. To build the module yourself instead:

git clone https://github.com/imqueue/redis-broker-unicaster.git
cd redis-broker-unicaster && make   # needs libuuid, libcurl, json-c
redis-server --port 6379 --loadmodule $PWD/unicaster.so

• DEPLOYMENT_ENV — the Kubernetes namespace to enumerate pods in (and therefore the blast radius of the announcements). Announcements reach only this namespace, so brokers and every service or client that should discover them must run in the same one. The name reads like an environment, but it is interpolated straight into /api/v1/namespaces/<value>/pods: set it from metadata.namespace as above rather than typing a value, because unset it requests /namespaces//pods, finds nobody, and reports nothing. The image refuses to start without it for exactly that reason.
• SELECTED_INTERFACES — comma-separated IP prefixes (e.g. 10.,192.168.) selecting which local interfaces announce themselves; unset means all of them, loopback included, so set it in real deployments.
• The RBAC side is a Role with list on pods plus a RoleBinding to the broker pod's service account.
• In the current implementation the announce destination port is fixed at 63000, so leave UDPClusterManagerOptions.port at its default on the service side.

The service side is the same in both recipes

Whatever transport the announcements take, services and clients configure one thing. A pattern that has served well in production keeps a static fallback one environment variable away:

const DISABLE_CLUSTER_MANAGER = !!+(process.env.DISABLE_CLUSTER_MANAGER || 0);
const cluster = (process.env.REDIS_CLUSTER || 'localhost:6379')
    .split(/\s*,\s*/)
    .map(cfg => {
        const [host, port] = cfg.split(/\s*:\s*/);
        return { host, port: +port };
    });

Object.assign(serviceOptions, DISABLE_CLUSTER_MANAGER
    ? { cluster }                                    // static list
    : { clusterManagers: [new UDPClusterManager()] } // discovery
);

One rule matters: apply the same cluster options to every service and every client. Requests and replies flow through the whole fleet, so a client pinned to a single broker will miss responses that round-robin landed elsewhere.

Encrypting the fleet

Everything above puts Redis traffic on the network in the clear. That is defensible inside a namespace you already trust, and indefensible the moment a compliance questionnaire asks about encryption in transit. Both halves are one setting each.

On the broker, mount a certificate:

docker run -v /path/to/tls:/run/tls:ro \
    -e IMQ_TLS_CERT_FILE=/run/tls/broker.crt \
    -e IMQ_TLS_KEY_FILE=/run/tls/broker.key \
    -e IMQ_TLS_CA_FILE=/run/tls/ca.crt \
    ghcr.io/imqueue/redis-broker:7.4

The TLS listener takes 6379 — the port the Service, the NetworkPolicy and the probes already name — so encrypting a fleet moves no ports and rewrites no manifests. Client certificates are required by default; IMQ_TLS_AUTH_CLIENTS=no encrypts without authenticating callers.

The announcement has to follow the listener. Redis serves TLS by setting port 0 and tls-port <n>, and the announcer modules used to advertise port verbatim — so a TLS broker announced 10.0.4.12:0, an address nothing can connect to and one that UDPClusterManager discards as malformed. The fleet discovered no broker at all, and nothing in any log said why: the announcement went out, it was just useless. From redis-broker v1.2.0 the modules advertise whichever listener is up and mark the datagram tls or plain. When both are up the plaintext port is announced, because that is what a running fleet is already connected to; REDIS_BROADCAST_TLS=1 picks the TLS port instead.

One certificate for the fleet, with no address in it. A broker takes the IP the scheduler hands it and announces that, so no certificate can name it in advance, and there is no DNS name to fall back on either — the fleet is found by announcement, not by lookup. Issue one certificate for the whole fleet carrying a name that will never be resolved, and have services pin it:

IMQ_REDIS_TLS_CA_FILE=/run/tls/ca.crt
IMQ_REDIS_TLS_SERVERNAME=imq-broker.internal   # compared, never resolved

servername is not a host to connect to: Node checks it against the certificate while the connection still goes to the announced IP. That is what lets a broker pod die and come back on a different address without anything being reissued — and it is the reason an auto-scaling fleet can be encrypted at all.

Turning it on for a fleet that is already running is a cutover rather than an overlap, because the announcement carries one transport for everybody: bring the brokers up with both listeners (IMQ_TLS_PLAINTEXT=on, which parks TLS on 6380 and leaves the announcement alone), then roll the services with their TLS options and the brokers with REDIS_BROADCAST_TLS=1 together, then drop both flags. The client half — one option covering every channel a queue opens, the IMQ_REDIS_TLS* environment fallback, mutual TLS, and what it costs — is a post of its own.

Life of the fleet

• A broker joins. Discovered within about one announce interval; @imqueue starts its queue, replays subscriptions, and folds it into the rotation. If a service sends before any broker is known (cold start), the send waits for the first discovery for up to 30 seconds (IMQ_SEND_INIT_TIMEOUT) instead of failing.
• A broker leaves gracefully. The module's shutdown hook emits down and removal is immediate. That's reliable at the default 1-second announce interval; at longer intervals shutdown can outrun the announcer thread, in which case removal falls back to heartbeat eviction.
• A broker crashes. No down arrives; the missed-heartbeat eviction removes it a few seconds later. Messages already queued on it stay in its Redis (subject to your persistence settings) and become consumable again when it returns — the fleet keeps flowing through the remaining brokers meanwhile.
• Auth. Give every broker the same credentials (one shared ACL file works well), because any service may connect to any discovered broker.
• Transport. The same rule, for the same reason: one transport for the whole fleet. The announcement carries a single host:port, so a broker serving TLS among neighbours serving plaintext is discovered and then unreachable by everything configured for the other one.
• Security. The datagrams are plain, unauthenticated UDP — anyone who can reach the port can inject or evict brokers. That is a deliberate trade with controls attached rather than an oversight, and it is worth stating in full rather than as a caveat: the next section does that.

What the announcement channel exposes

Anyone who can send a UDP datagram to port 63000 on the discovery address can announce a broker up, or announce a real one down. There is no signature and no shared secret; the datagram carries a name, a GUID, a status, a host:port and a transport marker, and any of them can be fabricated. Reaching the port is the whole of the attack.

Announcing a hostile broker puts an attacker-controlled address into every discovering client's rotation, so a share of real requests — arguments included — lands on it, and the replies their callers are waiting for never arrive. Announcing down evicts a real broker at once, and clients that have moved on miss the replies still in flight on it.

The amplifier is that announcements are not filtered by queue name. Every cluster registered with a manager on that address and port receives every announcement sent there — so two unrelated fleets sharing a segment and the default REDIS_BROADCAST_NAME discover each other's brokers with nobody attacking anything. The accident and the attack are the same mechanism, and the accident is far more likely.

Five controls, and the first is the one that matters:

1. A NetworkPolicy confining 63000/udp and 6379/tcp to the namespace.
2. A distinct REDIS_BROADCAST_NAME, and preferably port, per fleet.
3. SELECTED_INTERFACES pinned to the pod CIDR, so a broker never announces an address the fleet cannot reach.
4. RBAC scoped to list on pods in one namespace, for the unicaster.
5. TLS with client certificates on the brokers — which does not authenticate discovery either, but bounds what announcing a hostile broker is worth.

Why this is an acceptable design. The trust boundary is the namespace, and it is the same boundary that already protects Redis itself: an attacker who can send UDP to 63000 can almost always also open TCP to 6379, where without a password they can read every queued message and run FLUSHALL. Discovery does not add a perimeter — it sits inside the one you already have to defend. Setting a password does not change that either: it protects the data path, while the datagram stays unauthenticated, so it turns "can read your queues" into "can disrupt your routing".

TLS moves that line further without moving the boundary. With client certificates and a CA of your own, a broker announced at an attacker's address has to present a certificate signed by that CA before any service will send it a message — so announcing a hostile broker stops being a way to read traffic and becomes only a way to lose it. Announcing down is untouched: evicting a real broker needs no certificate. And the tls marker on the datagram is not a signal to trust — no client turns encryption on or off because of it, since that would let an unsigned UDP packet decide whether a connection is encrypted.

There is no signing or HMAC on the announcement today, and it is not planned: it would need a secret distributed to every broker and client and kept in step with the Redis password, for a payoff bounded by the port having to be reachable at all. If your threat model puts an untrusted party inside the namespace, use a static cluster: [...] list instead — you give up automatic scaling and gain a fleet that cannot be changed from the network. The full write-up, including the manifests, lives in THREAT-MODEL.md.

Picking a recipe

Environment	Recipe
Bare metal, on-prem VMs, one L2 segment	promoter (broadcast)
Docker bridge network, local development	promoter (broadcast)
Kubernetes — on GCP or any cloud VPC	unicaster (K8s-API unicast)
Cloud VMs outside Kubernetes, fixed topology	static cluster list, no modules

The broker fleet is the piece that turns "we can scale the services" into "the whole system auto-scales". If you're starting fresh, the getting-started guide gets a service and client running in minutes; for how the discovery mindset extends to services themselves, see do Node.js backends need service discovery? and load balancing without a load balancer.