08 — KEDA event-driven autoscaling
A Kafka consumer that scales 1 → 30 replicas based on consumer-group lag, plus VPA in Initial mode for right-sized memory requests on first pod start.
Next to it, an HTTP frontend that scales 0 → 10 replicas on request rate through the KEDA HTTP add-on, and wakes from zero on the first request.
What this shows
worker: Kafka lag
keda.enabled: trueon the worker, with akafkatrigger keyed on consumer-group lag.minReplicas: 1, not0— this worker should never have a gap in consumption. SetminReplicas: 0instead when the workload can tolerate sitting idle (e.g. acrontrigger active only during a scheduled window); KEDA scales it down to zero once every trigger goes inactive, and back up once one fires again.hpa.enabled: false(default) — KEDA itself manages an internal HPA; mixing both would be rejected by the chart’s mutual-exclusion guard.keda.advanced— passed into the ScaledObject verbatim.horizontalPodAutoscalerConfig.behaviorshapes the HPA that KEDA creates: one pod a minute up instead of the HPA default of doubling every 15 seconds, and 15 minutes of quiet before a replica is given back. Kafka lag is spiky, and the default behaviour overshoots a queue that drains fast.restoreToOriginalReplicaCount: trueputs the Deployment back on its pre-KEDA replica count if the ScaledObject is ever deleted. The chart validates the block as an object and nothing else, so any field in the ScaledObject spec works here.verticalPodAutoscaler.enabled: truewithupdateMode: Initial— VPA sets the resource request the first time each pod is created. Works alongside KEDA because VPA touchesrequests, KEDA touchesreplicas. (ADR 007)service.enabled: false— pure consumer, no inbound traffic.metrics.type: pod+integrations.monitoring.defaults.exposeService.enabled: true→ a metrics-only Service<release>-worker-metricsis auto-emitted, and aPodMonitorscrapes the worker pods directly.- The Deployment manifest will have no
spec.replicasbecause an HPA-class scaler is on — so Helmupgradedoesn’t bounce the count back to a static value on every sync.
web: HTTP traffic, down to zero
keda.http.enabled: trueturns the ScaledObject into one driven by the add-on’s interceptor, which counts requests, holds the first one while KEDA starts a pod, and forwards it once the pod is ready.minReplicasdefaults to0.- The Service keeps its name,
orders-web, and becomes anExternalNamefor the interceptor. The pods are behindorders-web-origin, which also carries the metrics port the ServiceMonitor scrapes. - The
httpport is left unset, so it takes the interceptor port,8080. An ExternalName cannot remap ports, so callers and the Ingress use8080. scalingMetric.requestRateadds one replica per 50 requests a second over a one-minute window.liveis a second port, for server-sent events.keda.http.routes.livegives it anInterceptorRouteof its own on/live:http://orders-web:8080/livereaches port9100, everything else reacheshttp. The route has its own trigger, so a stream request wakes the workload too. An open stream counts as one request for as long as it lasts, so itsconcurrencytarget is sized for connections.- The Ingress names
orders-webon8080. Its host,orders.example.com, is added to both InterceptorRoutes. podDisruptionBudget.maxUnavailable: 1replaces the root PDB for this workload:minAvailable: 1would block draining the node that runs its only pod.
A port that should not go through the interceptor goes under keda.http.directPorts instead and gets a Service of its own; its traffic neither counts nor wakes the workload. The full set of fields, and the shapes the render rejects, are in the values reference.
Delta from 02-web-app-ingress
| Changed | What |
|---|---|
| HPA → KEDA | Event-driven scaling; HPA off, KEDA on. |
| Scaling behaviour | keda.advanced.horizontalPodAutoscalerConfig.behavior caps the ramp at one pod a minute. |
| Added VPA | Orthogonal, recommends/sets memory requests on initial pod create. |
| Service off | Worker has no listener; metrics-only Service auto-emitted instead. |
metrics.type: pod | PodMonitor instead of ServiceMonitor. |
Ingress on web | Backend is the ExternalName on the interceptor port. |
web on KEDA HTTP | Scales to zero; the interceptor wakes it on the first request. |
Files
| File | Purpose |
|---|---|
values.yaml | Chart values. |
argocd/application.yaml | Argo CD Application, single source. |
argocd/application-multisource.yaml | Argo CD Application with spec.sources[]. |
flux/ocirepository.yaml | Flux OCIRepository. |
flux/helmrelease.yaml | Flux HelmRelease. |
helm/install.sh | Plain helm upgrade --install. |
Try it
bash helm/install.sh
kubectl apply -f argocd/application.yaml -n argocd
kubectl apply -f flux/
Prerequisites
- KEDA installed in the cluster (
keda.shCRDs). - KEDA HTTP add-on v0.14.0 or later in the
kedanamespace (http.keda.sh/v1beta1InterceptorRoute). Setintegrations.keda.httpAddonif it runs elsewhere. - An Ingress controller that accepts
ExternalNamebackends; ingress-nginx does. - VPA installed (
autoscaling.k8s.ioCRDs). - A Kafka cluster reachable at
kafka.kafka:9092with topicordersand consumer grouporders-worker. Adjust to your environment. - Prometheus Operator CRDs for the PodMonitor (or remove the
metrics:block).