How to Design Health Checks That Find Real Failures

A service returns “OK” from its health endpoint while every checkout request fails. The process is alive, but the user journey is broken. Health checks become useful when each check has a specific question, a defined consumer, and an appropriate consequence for failure.

Separate the questions

A startup check asks whether initialization has completed. A liveness check asks whether restarting this process could resolve its condition. A readiness check asks whether the instance should receive traffic now. A synthetic transaction asks whether a representative user operation works across the real route and its dependencies.

Kubernetes documents these distinctions and their effects in its probe guidance. A failed readiness probe can remove a Pod from service traffic; repeated liveness failures can lead to restarts. The consequences matter as much as the endpoint name. Configure the check for the action you want the platform to take.

Do not make a downstream database outage automatically restart every application instance. Restarting healthy processes does not repair the database and may increase pressure during recovery. Decide whether the service can provide limited functionality or should temporarily stop accepting dependent requests.

Choose a minimal, meaningful liveness signal

Keep liveness local and cheap. A process responding on its normal request path may demonstrate more than a separate thread that always returns success while workers are stuck. The right signal depends on the runtime and failure mode. Explain why a restart is likely to help when the signal fails.

Avoid expensive queries, external API calls, and broad filesystem scans in a frequently repeated probe. Probes compete with user traffic. During overload, an expensive check can worsen the condition it is trying to observe. Use bounded work and timeouts even for internal diagnostic requests.

Provide enough startup allowance for normal initialization without hiding a permanent startup failure. A slow migration, cache warmup, or dependency negotiation can make a fixed aggressive liveness threshold destructive. Prefer an explicit startup boundary where the platform supports it.

Make readiness represent service capacity

Readiness can include required local initialization and dependencies essential to the routes this instance serves. Be careful about shared dependencies: marking every instance unready because a nonessential analytics service is unavailable may remove a functioning application from traffic unnecessarily.

Decide how quickly readiness should change and recover. Use thresholds that avoid flapping on isolated delays while still reacting within an acceptable interval. Record these values as choices tied to the application's behavior, not universal defaults that every service should copy.

When draining traffic for deployment, coordinate readiness with connection handling and shutdown. Stopping new requests is different from immediately killing active ones. Long-running requests, background jobs, and persistent connections may need separate termination behavior.

Observe the user path separately

A public check should traverse the route users depend on, including DNS, TLS, routing, and relevant application behavior. An internal endpoint cannot establish that an expired public certificate or broken proxy configuration is harmless. Use a harmless synthetic account or fixture where a real operation is necessary.

Google's monitoring guidance distinguishes externally visible behavior from internal instrumentation and discusses latency, traffic, errors, and saturation. Combine these perspectives. A synthetic failure tells you something is wrong; internal signals help locate the cause.

Keep synthetic operations safe. Do not create real payments, spam customers, or accumulate permanent test records simply to prove a route works. If a check writes data, give it a distinct identifier, limited permissions, and a cleanup plan. A read-only check may be appropriate but should not be presented as proof that writes succeed.

1. Choose the operational question and failure response.
2. Select the smallest bounded signal that answers it.
3. Define startup allowance, timeout, and repeated-failure behavior.
4. Exercise the intended failure in a controlled environment.
5. Pair the check with user-path monitoring and a runbook.

Hypothetical example: a booking API

In this hypothetical example, a booking API loads configuration, connects to its database, and accepts requests through a reverse proxy. Startup remains incomplete until required initialization finishes. Liveness checks local responsiveness. Readiness determines whether the instance can accept booking traffic.

The team also plans a public synthetic check that reads availability for a dedicated test listing. That check can reveal routing or certificate failures, but it does not prove that a customer can complete a paid booking. A separate controlled test is needed for that path, with safeguards around external payment effects.

When the database becomes unavailable, the team expects dependent routes to fail clearly and readiness to reflect the intended service policy. It does not assume a restart will fix the dependency. The scenario describes a design to validate, not observed production behavior.

Keep diagnostic output safe

A public health response rarely needs hostnames, credentials, stack traces, or complete dependency details. Return a minimal result to the public consumer and keep detailed diagnostics behind appropriate access. Logs should connect failures to a trace or instance identifier without exposing secrets.

Every actionable alert needs an owner and an initial response. If a check repeatedly fails without anyone knowing what to do, improve the check or its runbook. Do not simply increase its threshold until the dashboard becomes quiet.

Health-check checklist

• Does each probe answer one named operational question?
• Is its failure consequence appropriate and documented?
• Could a dependency outage trigger an unhelpful restart storm?
• Are probe work and response time bounded?
• Are startup and graceful shutdown handled explicitly?
• Does monitoring include the actual public route?
• Are synthetic operations safe and diagnostic details protected?

Review checks when the service's responsibilities change. A probe that once represented the critical path can become irrelevant after a new dependency or route is introduced. Keep the evidence aligned with what users actually need to do.