KEDA vs HPA for RabbitMQ Consumers

CPU looks fine while the queue drowns

Monday morning traffic dumps forty minutes of outbox events into RabbitMQ after a weekend maintenance window. Three consumer pods sit at twenty-two percent CPU. The queue depth climbs past one hundred twenty thousand messages. Downstream projections lag eighteen minutes behind real time and the support desk starts forwarding “stale dashboard” tickets. Horizontal Pod Autoscaler never adds a replica because it only watches CPU. The autoscaler is doing its job; it was given the wrong signal.

Problem in routing and scaling terms

.NET worker services that consume from RabbitMQ queues are I/O-bound for much of their lifecycle: deserialize JSON, call HTTP dependencies, write rows, acknowledge messages. CPU stays flat while queue depth rises. Kubernetes HPA v2 defaults encourage resource metrics. Without custom metrics adapters, teams scale on CPU or memory and wonder why lag correlates with business hours rather than pod count.

Figure 1 shows the failure mode: messages accumulate while HPA sees idle CPU.

sequenceDiagram
    participant Pub as Outbox relay
    participant RMQ as RabbitMQ queue
    participant C1 as Consumer pod
    participant HPA as Kubernetes HPA
    participant KEDA as KEDA ScaledObject
    Pub->>RMQ: Publish 50k messages
    RMQ->>C1: Deliver (prefetch 10)
    Note over C1: CPU 22%, handler waits on HTTP
    HPA->>HPA: cpu < 70% → no scale
    Note over RMQ: depth 120k, lag 18m
    KEDA->>RMQ: Read queue length
    KEDA->>C1: Scale to 12 replicas

KEDA (Kubernetes Event-driven Autoscaling) reads external signals such as RabbitMQ queue length and adjusts replica count on ScaledObject resources. HPA remains useful for HTTP APIs where CPU tracks load. For queue consumers, unconsumed work is the honest metric.

Why vendor defaults mislead

The Kubernetes HPA documentation describes scaling on resource and custom metrics. RabbitMQ queue depth is not a built-in metric. Platform tutorials that deploy a Deployment plus HPA on CPU copy cleanly into production and fail silently under backlog. RabbitMQ’s own guidance on consumer utilisation emphasises prefetch, ack rates, and queue monitoring rather than container CPU.

KEDA’s RabbitMQ queue scaler documents authentication modes, queue length targets, and activation thresholds. Teams skip reading activation thresholds and see flapping when depth hovers near zero; that is configuration, not a scaler limitation.

Decision table: HPA vs KEDA by signal

Workload signal	HPA (CPU/memory)	KEDA (queue depth)	Notes
ASP.NET HTTP API	Fit	Rare	CPU/memory track RPS when requests are CPU-heavy
RabbitMQ consumer (.NET worker)	Poor	Fit	Scale on messages ready + unacked strategy
Scheduled batch spike	Poor	Fit	Depth rises before CPU; scale ahead of SLA breach
Idle queue overnight	Over-provisioned	Scale to zero optional	Set minReplicaCount 0 with activation threshold
Poison message loop	Misleading	Dangerous	Pair with DLQ alerts; scaling amplifies bad traffic

Architecture rule: Scale message consumers on unconsumed work (queue depth or age), not on container CPU, unless profiling proves CPU saturation precedes backlog.

Production-shaped KEDA configuration

ScaledObject for a quorum queue fed by an outbox relay:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: outbox-consumer
spec:
  scaleTargetRef:
    name: outbox-consumer
  minReplicaCount: 2
  maxReplicaCount: 24
  pollingInterval: 15
  cooldownPeriod: 120
  triggers:
    - type: rabbitmq
      metadata:
        protocol: amqp
        queueName: outbox.relay
        mode: QueueLength
        value: "500"
        activationValue: "50"
      authenticationRef:
        name: rabbitmq-trigger-auth

value: "500" targets roughly five hundred messages per replica before adding capacity. Tune with measured handler throughput: if one pod sustains eighty messages per minute at p95 dependency latency, five hundred messages implies about six minutes of work per pod, which is a reasonable buffer before lag alarms fire.

Consumer-side settings belong in the same change ticket:

{
  "RabbitMq": {
    "PrefetchCount": 20,
    "ConsumerDispatchConcurrency": 4
  }
}

Raising replicas without adjusting prefetch can overload downstream SQL pools. Scale consumers and database connection limits together.

Implementable checklist

1. Export queue depth, publish rate, ack rate, and consumer utilisation to Grafana; alert on depth and message age, not CPU.
2. Install KEDA in the cluster; verify ScaledObject can read RabbitMQ management API or AMQP metadata.
3. Remove CPU HPA from the consumer Deployment or set HPA max below KEDA max only when CPU is a secondary cap.
4. Load-test with fifty thousand synthetic outbox messages; assert replica count rises within ninety seconds and lag returns under SLA within ten minutes.
5. Document poison-message playbooks: DLQ depth alarm pauses ScaledObject max or switches traffic to a quarantine queue.
6. Run chaos on a single pod kill; verify messages requeue and depth-driven scale replaces capacity.

Prefetch, quorum queues, and scaler interaction

Classic queues with aggressive prefetch hide backlog from the management API: messages sit unacked on slow consumers while ready count looks healthy. Prefer quorum queues for outbox relays when you need predictable behaviour under node loss; pair KEDA’s QueueLength mode with monitoring on unacked message age. If your cluster still runs classic mirrored queues, document the migration ticket before tuning scalers, because depth semantics differ during partial ack stalls.

Consumer concurrency in .NET 8 worker services (ConsumerDispatchConcurrency) multiplies in-flight handlers per pod. KEDA replica count × prefetch × concurrency is the effective parallelism budget. Example: twelve replicas, prefetch twenty, concurrency four yields up to nine hundred sixty unacked deliveries. If your SQL pool allows one hundred connections, you have a predictable outage unless handlers release connections before ack.

Activation thresholds prevent cold-start flapping when depth oscillates between zero and five messages overnight. Set activationValue high enough that a single pod handles idle traffic without scaling to zero unless cost savings justify wake latency. Scale-to-zero saves money on dev clusters; production outbox relays usually keep minReplicaCount: 2 for zone redundancy regardless of KEDA’s ability to wake from zero.

Observability that catches the next mis-scaling

Dashboards should plot queue depth, consumer count, ack rate, publish rate, and handler p95 in one row. Alert when depth rises while replica count is flat for more than three polling intervals. That pattern means KEDA is misconfigured, RBAC blocks metric reads, or HPA is fighting KEDA with a lower max. Log scaler decisions at info level during rollout week; operators should see replica changes correlate with depth spikes in timestamps, not minutes later.

Run game days: pause all consumers for two minutes, resume, and verify KEDA adds capacity before lag alarms fire. Repeat with a single availability zone cordoned off. Autoscaling policy is infrastructure code; test it like a failover script, not like a Helm chart checkbox.

Ops gates and sign-off

SLO: p95 end-to-end lag from outbox insert to downstream read model under five minutes at P99 publish burst of ten thousand messages per minute. Incident class B when lag exceeds fifteen minutes for more than five minutes. Platform signs off KEDA thresholds; application team signs off prefetch and handler concurrency; joint review quarterly or after any RabbitMQ cluster upgrade.

HTTP APIs and queue workers rarely belong on the same scaling policy. Split Deployments, split metrics, split runbooks. Teams building event-driven .NET platforms on Kubernetes often need both patterns in one estate; getting the boundary wrong is cheaper to fix in design review than in a Monday backlog.

For architecture reviews on RabbitMQ relays, KEDA scaling, and .NET consumer hardening, our custom software development team documents the metric map before any Helm chart changes ship to production.