Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

08 — KEDA event-driven autoscaling

A Kafka consumer that scales 1 → 30 replicas based on consumer-group lag, plus VPA in Initial mode for right-sized memory requests on first pod start.

What this shows

  • keda.enabled: true on the worker, with a kafka trigger keyed on consumer-group lag.
  • minReplicas: 1, not 0 — this worker should never have a gap in consumption. Set minReplicas: 0 instead when the workload can tolerate sitting idle (e.g. a cron trigger active only during a scheduled window); KEDA scales it down to zero once every trigger goes inactive, and back up once one fires again.
  • hpa.enabled: false (default) — KEDA itself manages an internal HPA; mixing both would be rejected by the chart’s mutual-exclusion guard.
  • keda.advanced — passed into the ScaledObject verbatim. horizontalPodAutoscalerConfig.behavior shapes the HPA that KEDA creates: one pod a minute up instead of the HPA default of doubling every 15 seconds, and 15 minutes of quiet before a replica is given back. Kafka lag is spiky, and the default behaviour overshoots a queue that drains fast. restoreToOriginalReplicaCount: true puts the Deployment back on its pre-KEDA replica count if the ScaledObject is ever deleted. The chart validates the block as an object and nothing else, so any field in the ScaledObject spec works here.
  • verticalPodAutoscaler.enabled: true with updateMode: Initial — VPA sets the resource request the first time each pod is created. Works alongside KEDA because VPA touches requests, KEDA touches replicas. (ADR 007)
  • service.enabled: false — pure consumer, no inbound traffic.
  • metrics.type: pod + integrations.monitoring.defaults.exposeService.enabled: true → a metrics-only Service <release>-worker-metrics is auto-emitted, and a PodMonitor scrapes the worker pods directly.
  • The Deployment manifest will have no spec.replicas because an HPA-class scaler is on — so Helm upgrade doesn’t bounce the count back to a static value on every sync.

Delta from 02-web-app-ingress

ChangedWhat
HPA → KEDAEvent-driven scaling; HPA off, KEDA on.
Scaling behaviourkeda.advanced.horizontalPodAutoscalerConfig.behavior caps the ramp at one pod a minute.
Added VPAOrthogonal, recommends/sets memory requests on initial pod create.
Service offWorker has no listener; metrics-only Service auto-emitted instead.
metrics.type: podPodMonitor instead of ServiceMonitor.
Removed IngressNo external traffic.

Files

FilePurpose
values.yamlChart values.
argocd/application.yamlArgo CD Application, single source.
argocd/application-multisource.yamlArgo CD Application with spec.sources[].
flux/ocirepository.yamlFlux OCIRepository.
flux/helmrelease.yamlFlux HelmRelease.
helm/install.shPlain helm upgrade --install.

Try it

bash helm/install.sh
kubectl apply -f argocd/application.yaml -n argocd
kubectl apply -f flux/

Prerequisites

  • KEDA installed in the cluster (keda.sh CRDs).
  • VPA installed (autoscaling.k8s.io CRDs).
  • A Kafka cluster reachable at kafka.kafka:9092 with topic orders and consumer group orders-worker. Adjust to your environment.
  • Prometheus Operator CRDs for the PodMonitor (or remove the metrics: block).