DevOps
Chaos Engineering
Deliberately injecting failure — killed instances, added latency, severed dependencies — to test resilience claims before real outages test them for you. Netflix's Chaos Monkey made it famous. It only pays off once the basics exist (monitoring, retries, fallbacks), and every experiment needs a blast radius you control.
Purpose
Every system carries untested resilience claims: "if the cache dies we fall back to the database", "retries handle transient failures". Chaos engineering tests them empirically — define the steady state, hypothesise it survives a specific failure, inject that failure deliberately, and watch. The point is not breaking things; it is discovering that the fallback misbehaves during a controlled Tuesday experiment instead of a real Saturday outage.
When to Use It
Verifying that instance loss is a non-event (the original Chaos Monkey), that timeouts and circuit breakers actually trip at the configured thresholds, that a degraded dependency degrades you gracefully rather than totally, and that on-call runbooks match reality — game days, where a staged incident exercises both the system and the humans.
Trade-offs
It is a maturity-gated practice: injecting failure into a system without monitoring or fallbacks just causes outages with extra steps. Experiments carry real risk, so blast radius control — staging first, then a sliver of production, with an abort switch — is non-negotiable. And the experiments themselves are infrastructure to build and maintain; the cost is only worth it where downtime is expensive.
Implementation
Start with a hypothesis ("p99 stays under 200ms if one replica dies"), the smallest useful blast radius, and a rollback trigger. Inject one failure class at a time — instance kill, latency, dependency error (tools: Chaos Mesh, Gremlin, AWS FIS, or tc netem) — compare against the steady state, and fix what surprised you before escalating scope. Automate the experiments that prove invariants worth re-proving on every release.