Icon
Sreeram Venkitesh
Published on

How to run Kubernetes using kind with a custom build of containerd

586 words • 3 min read
#kubernetes#kubernetes-dev#open-source

Contents

  1. Compile containerd
  2. Build Kubernetes
  3. Create a new kind image overriding the container runtime
  4. Create your cluster
  5. Appendix: Linking CRI API with containerd

In one of my previous posts in this series, I wrote about the different ways in which you can customize kind to run Kubernetes in your machine. I had written about the different ways you can write your kind configuration to setup multi node clusters, simulate version skew between different nodes and how can have total fine grained control over all the feature gates of all the different components.

If you're working with the Container Runtime Interface (CRI) API, you would want to run your kind cluster with a particular container runtime or a particular version of a container runtime. You might also want to test changes you make to a container runtime by running Kubernetes with a container runtime that has you changes. Here are the instructions on how you can do this:

Compile containerd

Once you've made the changes to containerd, compile it to get the binary

# from inside the containerd directory
GOOS=linux GOARCH=arm64 make

I'm passing the environment variables GOOS=linux GOARCH=arm64 since I've been using M series Macbooks with arm architecture.

Build Kubernetes

If you want to test changes you've made to the Kubernetes codebase along with your changes to containerd, you can build Kubernetes as well. If you only have changes to containerd, you can skip this step and use an existing image like kindest/node:v1.32.0 instead.

kind build node-image --arch arm64 .

Create a new kind image overriding the container runtime

Create the following Dockerfile where you use the Kubernetes image you built (or the pre-existing Kubernetes image you pulled) as the base image and copy the containerd binary you just compiled into it.

FROM kindest/node:latest

COPY ./bin/containerd /usr/local/bin/containerd

RUN chmod +x /usr/local/bin/containerd

Build the image like so:

docker build -t kindest/node:custom-containerd .

Now you will have a new kindest/node:custom-containerd image which will use your build of containerd instead of the one that comes by default with kind.

Create your cluster

Create a cluster using the image you just built.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:custom-containerd
- role: worker
  image: kindest/node:custom-containerd

Once you start the cluster, you can exec into the nodes and run journalctl -u containerd to see the container runtime logs.

Appendix: Linking CRI API with containerd

If your changes to containerd involve using changes from the CRI API, you need to let containerd know to use the CRI API package that you have updated locally. You need to do this because the version of the CRI API package in your local Kubernetes repo you're working with is not released yet. You can do this by adding the following line to the go.mod file in containerd before you compile it.

replace k8s.io/cri-api => /path/to/your/local/cri-api

After you've added this line, run go mod tidy:

go mod tidy
go mod vendor
GOOS=linux GOARCH=arm64 make # You'd need to build containerd again

This would make containerd use the cri-api package from your local Kubernetes project. For example if you want to change the parameters of the StopContainer function and make it take a new StopSignal field from the ContainerSpec which Kubernetes sends over through the CRI API, you'd need to do the following:

  • Update the CRI API in Kubernetes
  • Update containerd to use your version of the cri-api package
  • Update the StopContainer function in containerd to use the data sent over by Kubernetes in the new ContainerSpec, which now you've updated to include StopSignal as well.