How to Retry Like a Boss

Today I learned How to retry like a boss. And the good part – K8s already did it for you.

func OnError(backoff wait.Backoff, retriable func(error) bool, fn func() error) error

Simple code that allows you to run fn and retry over and over until a certain condition occurs. let’s say you want to poll for your remote service and sometimes this poll call fails:

func IsServiceStatusOk(addr string) (bool, error) {
    svc := NewServiceConnection(addr)
    status, err := svc.GetStatus()
    if err != nil {
        return false, err
    }
    return status == statuscodes.Ok, nil
}

it may fail due to a bad status code, or due to an error in svc.GetStatus() call. Let’s just say that whatever error you are getting you’ll want to retry it, except for io.ErrClosedPipe – saying you need to re-open the connection so there is no use in trying again. here is a translation to code:

func shouldRetry(addr string) (err error) bool {
    if err == io.ErrClosedPipe {
        return false    
    }
    return true
}

the next step is to decide what is your retry strategy. You don’t want to blast the service with status checks it will consume the CPU for you and might cause DoS for the other side. The default backoff strategy of k8s is this:

var DefaultRetry = wait.Backoff{
	Steps:    4,
	Duration: 10 * time.Millisecond,
	Factor:   5.0,
	Jitter:   0.1,
}

So the final retry code will look something like this

retry.OnError(retry.DefaultBackoff,shouldRetry,func() error {
    ok, err := IsServiceStatusOk(myAddr)
    if err != nil {
        return err
    }
    return fmt.Errorf("status is not ok")
})

That’s it! for further reading see:
Kubernetes retry go package: https://pkg.go.dev/k8s.io/client-go/util/retry
Kubernetes wait package: https://pkg.go.dev/k8s.io/apimachinery/pkg/util/wait