Testing your Code on Kubernetes

It can sometimes be a bit cumbersome to actually test your code on Kubernetes proper, as there are many moving parts involved. You need to compile the code, build helm charts, build and push the container images, etc.

While we do have CI actions for this, those actions also run some tests and other checks that can take a fair bit of time and are not readily available to developers outside of the Stackable organization. Also, if you need to make changes to an operator and the framework at the same time, compiling the operator with your local version of the framework becomes very difficult in CI actions.

For these reasons we have created a developer focused deployment mechanism that allows for easy local development, while still enabling full-scale testing in an actual Kubernetes cluster.

The main tool that is used for enabling these short feedback loops is called Tilt. Tilt is a tool that continuously monitors your local codebase and automatically deploys any changes you make to the Kubernetes cluster defined by your current kubeconfig.

Effectively this means, that when you have reached a state in your code that you would like to deploy to Kubernetes to look at more in depth, all you need to do is .. nothing - it has already been built, packaged and deployed in the background.

A very important prerequisite for this of course is short build times!

To shorten these, we have settled on a tool called crate2nix. This tool uses the Nix package manager to cache intermediate build steps and only recompile what has actually changed, thus significantly shortening build times.

Installation

Due to the nature of how Nix works, all the setup steps are defined in the operator repositories and automatically applied when you start using this workflow.

The only prerequisite you need to install is the actual Nix package manager - you can find installation instructions and additional documentation on the Nix website.

TL/DR

sh <(curl -L https://nixos.org/nix/install) --daemon

If you don’t want to run an arbitrary shellscript directly from the web, have a look at how to install from a binary distribution or the list of maintained packages.

After this is done you also need to add a setting to your Nix config in /etc/nix/nix.conf:

experimental-features = nix-command

Just installing Nix does not affect your system much, as it keeps all its configuration and installed packages separate from other package managers and you won’t even notice it is there, unless you actually start using it.

Using

The build and deploy steps for installing and running the operator are defined in the Tiltfile in the operators repository. We do encourage you to check out this file if you are interested in how things work under the hood, but you can also just use the command provided below and everything should just work. For more context on how to read this file please have a look at the Tiltfile API Reference, which is based on Starlark.

We provide a target in the Makefile to start everything up:

make run-dev

After running this, Tilt should be up and doing its thing, in the console you’ll see something similar to the following:

➜  hdfs-operator git:(main) ✗ make run-dev
nix run -f. tilt -- up --port 5430
Tilt started on http://localhost:5430/
v0.30.13, built

(space) to open the browser
(s) to stream logs (--stream=true)
(t) to open legacy terminal mode (--legacy=true)
(ctrl-c) to exit

You can now either hit the spacebar to open the Tilt user interface, or manually navigate to the url shown.

The port used will be different for every repository from the Stackable organisation, in order to allow running multiple deployment workflows at the same time without getting port conflicts.

Configuring the Registry Used

If you are using a local Kubernetes like Kind, K3s or similar for your development Tilt will work right out of the box for you, as it will directly push the images to your local Kubernetes cluster (see https://docs.tilt.dev/personal_registry.html for more information).

Due to the way that images are pushed to Kind this can be fairly inefficient, as the whole image will need to be pushed to every Kind node every time, not just changed layers once. To work around this, Kind can be set up to use a local registry, to which Tilt can then push the images. The easiest way that we found to do this is by using ctlptl which enables you to easily set up local Kind, K3s or minikube clusters with a local registry. Tilt should then automatically discover this registry from the cluster config and push images there.

If you are using a remote cluster, Tilt will push the generated container images to a remote registry, in order for your Kubernetes to be able to access them. We have configured docker.stackable.tech/sandbox as the default registry, as this is what all our developers use. External contributors will not have access to this registry and need to override this to a registry of their choice.

Overriding the default registry can be done by providing a file called tilt_options.json in the same directory as the Tiltfile.

{
"default_registry": "docker.stackable.tech/soenkeliebau",
}