Today I learned how to release a new version for dev testing faster. By leveraging external tools, and installing in-cluster components you can enhance your dev cycle, deploy new versions at the speed of light, and, eventually, bring new features in less time.
Usually, the development cycle consists of shorter and longer cycles. You start with writing several functions and then check the basic operation with a unittest or two. Your next step will probably be to create a version of it. But is not always enough. Your product might not easily be tested as a binary or in a stand-alone run through the IDE.
And anyway, you will want eventually to test it on your dev cluster. The environment in a Kubernetes cluster is different than your local laptop. Sometimes the liveness and readiness probes will fail with the new feature. That is if the cluster managed to even start your new image. These are the things you want to make sure to be working before creating a PR.
About the Dev Cycle
When you unittest your function at the start of the developing process, you encounter a lot of errors and issues. Undeclared variable, wrong return type, missing dependency… All can be solved easily with a slight change, and to rerun the test. The period from changing this small line to the feedback of the other error from the test I call the “dev cycle”. At the beginning of the feature, it is short since we check it with a simple unittest that compiles within a second. But later, when a deployment to a cluster is needed… You can sit back and relax as running it through the build pipeline can take more than a few minutes.
The Solution
In order to make this cycle shorter and faster, you can choose from many solutions. One of them is to use Skaffold. It handles the workflow for building, pushing, and deploying your application to Kubernetes. But for some use cases, it may be complex to use. For example, I’m my workplace we have a very specific upgrade strategy that cannot be copied with Helm or kubectl. Or when working with several components it makes the initialization of Skaffold painful.
The other solution is to run a container registry inside the cluster. Which have a lot of pros:
- You don’t need permission to push. Sometimes companies only allow formal pipelines to push so this overrides this issue.
- You leverage the build-cache as you can push locally-built images.
- Save money, as you store fewer images (do you ever delete old and stale images from the company registry? I don’t think so…).
- Faster push and way faster pull as the bandwidth to your cluster is usually larger and pulling from inside the cluster is lightning-fast.
- Unlimited API queries. Usually, dockerhub allows 100-200 pulls per 6 hours or 5000 if you have paid account. The local registry does not have this limit.
- you can use whatever name you want for images and tags.
Let’s start! In order to create a local registry we will need several components: a namespace (optional), a deployment, Volume (persistent volume claim), and a service. I used bare-metal Kubernetes and a virtual (kind/docker) one so if you want to host it in your cloud provider things might be different.
apiVersion: v1
kind: Namespace
metadata:
name: docker-registry
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: registry-pvc
namespace: docker-registry
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: registry
namespace: docker-registry
spec:
replicas: 1
selector:
matchLabels:
app: registry
template:
metadata:
labels:
app: registry
spec:
containers:
- name: registry
image: registry:2
ports:
- containerPort: 5000
volumeMounts:
- name: registry-storage
mountPath: /var/lib/registry
volumes:
- name: registry-storage
persistentVolumeClaim:
claimName: registry-pvc
---
apiVersion: v1
kind: Service
metadata:
name: registry
namespace: docker-registry
labels:
app: registry
spec:
type: NodePort
ports:
- port: 5000
targetPort: 5000
nodePort: 30500
selector:
app: registry
in order to push you’ll need to tag your images with <clusterIP>:30500/<image_name>:<image_tag> and pulling from inside the cluster should be the same.
A final note: your deployment may vary and you’ll want to use a load-balancer instead of a node port, or you might have an HTTPS issue. But trust me, invest a few minutes to resolve these issues once, and you’ll gain back hundreds of hours back. It made my deployment 10 times faster. From 10-20 minutes to 1-2.
Further Reading
Skaffold – https://skaffold.dev/
Docker registry image – https://hub.docker.com/_/registry
Adding TLS certificate – https://docs.docker.com/registry/deploying/#run-an-externally-accessible-registry
Remove images from the registry – https://gist.github.com/jaytaylor/86d5efaddda926a25fa68c263830dac1