Today I learned about how Liveness and Readiness probes work in Kubernetes. Not everyone knows, but there are actually 3 available probes in Kubernetes: liveness, readiness, and startup. Each has its own reason and effect for success or failure. And, most important, have best practices for how and what to use.
Why
The importance of health checks in Kubernetes is important for two reasons. One- to make sure we manage to load a new pod successfully. Tow- in order to make sure our application runs smoothly, and auto-remediate some fatal cases. If you want to keep your application service at top-grade and make the deployment long-lasting. I REALLY recommend sticking to these best practices. Don’t try to workaround the probes to your case, adapt your case to the probes.
What
I believe this table should explain best the difference between the actions of the probe:
| use-case | Liveness | Readiness | Startup |
| What happens after initial success | Nothing | The pod starts to receive external traffic | Liveness and Readiness probes start to take action |
| What happens if the probe never succeded | Pod will restart | Nothing | Pod will restart |
| What happens if the probe succeeded and later on it fails | Pod will restart | The pod will stop receiving new traffic | Does not apply |
To sum up, the recommended checks for each probe are:
- Liveness– to check up on dead-locks in your running app. most people actually get this probe wrong by checking if they can get a simple HTTP response from their app. This is not the correct way because usually HTTP servers run in a different thread and this does not assure that your program is not stuck somewhere else in your code.
- Readiness – to signal that your application is prepared to serve traffic. This means you manage to initiate all required dependencies including populating internal memory and successfully connected to external services such as DB or remote backend service. This is also the reason why you should report back when an external connection was broken.
- Startup – from Kubernetes’ official documentation: “Sometimes, you have to deal with legacy applications that might require an additional startup time on their first initialization. In such cases, it can be tricky to set up liveness probe parameters without compromising the fast response to deadlocks that motivated such a probe.“
How
As I mentioned, each probe covers a different aspect and has different results on success or failure. But they are all being checked from the Kubernetes side in the same way.
The execution of the check can be done by running a command in your container (which should not be available if you run from scratch) or using a network port (TCP, HTTP, or even gRPC from Kubernetes 1.23+).
In terms of timing, you can control the initial delay where the probe is not being checked, the regular checking period in seconds, and even the probe timeout.
And sometimes you want to tweak the success or failure criteria. You might don’t want that a single error will terminate your running container.
For further reading:
Container Probes: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
Probes definitions on Kubernetes documentation: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#Probe