Kubernetes v1.25 introduced Container Checkpointing API as an alpha feature. This provides a way to backup-and-restore containers running in Pods, without ever stopping them.
This feature is primarily aimed at forensic analysis, but general backup-and-restore is something any Kubernetes user can take advantage of.
So, let's take a look at this brand-new feature and see how we can enable it in our clusters and leverage it for backup-and-restore or forensic analysis.
Setup
Before we start checkpointing any containers, we need a playground where we can mess with kubelet and its workloads. For that we will need a v1.25+ Kubernetes cluster and container runtime that supports container checkpointing.
We will create such cluster using kubeadm inside VM(s) built with Vagrant. I've created repository with everything necessary to spin up such cluster with just vagrant up, so if you want to follow along, do check it out.
If you want to build your own cluster, then make sure it satisfies the following:
The cluster must have ContainerCheckpoint feature flag enabled. For kubeadm use following configuration:
# kubeadm-config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
ContainerCheckpoint: true
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.25.0
apiServer:
extraArgs:
feature-gates: "ContainerCheckpoint=true"
controllerManager:
extraArgs:
feature-gates: "ContainerCheckpoint=true"
scheduler:
extraArgs:
feature-gates: "ContainerCheckpoint=true"
networking:
podSubnet: 10.244.0.0/16
This will pass the --feature-gates flag to each of the cluster components. For full list of available feature gates see docs.
Additionally, we will also need to use a container runtime that supports checkpointing. At the time of writing only CRI-O supports it, with containerd probably coming soon-ish.
To configure your cluster with CRI-O, install it using instructions in docs, or use the convenience script in above-mentioned repository (you should run this in VM, not on your local machine).
Additionally, we need to enable CRIU for CRI-O, which is the tool that does the actual checkpointing in the background.
To enable it, we need to set --enable-criu-support=true flag. The above convenience script does that for you.
Also, if you plan to restore it back into a Pod, you will also need --drop-infra-ctr set to false, otherwise you will get CreateContainerError with message like:
kubelet Error: pod level PID namespace requested for the container, ...
... but pod sandbox was not similarly configured, and does not have an infra container
With CRI-O installed, we also have to tell kubeadm to use its socket, the following configuration with take care of that:
# kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.56.2
bindPort: 6443
nodeRegistration:
criSocket: "unix:///var/run/crio/crio.sock"
---
# ... the previous snippet goes here
# Full config at:
# https://github.com/MartinHeinz/kubeadm-vagrant-playground/blob/container-checkpoint-api/kubernetes/kubeadm-config.yaml
With that in place we can spin-up cluster with:
kubeadm init --config=.../kubeadm-config.yaml --upload-certs | tee kubeadm-init.out
This should give us a single node cluster such as (note the container runtime version):
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION ... OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubemaster Ready control-plane 82s v1.25.4 ... Ubuntu 20.04.5 LTS 5.4.0-125-generic cri-o://1.25.0
Note: Generally, the best and simplest way to play with Kubernetes is using KinD. KinD however (as of time of writing) doesn't support container checkpointing. Alternatively you can also try local-up-cluster.sh script in Kubernetes repository.
Checkpointing
With that out of the way, we can try creating a checkpoint. The usual operations on Kubernetes can be done with kubectl or by running curl commands against cluster API server. This however won't work here, as the checkpointing API is only exposed on the kubelet on each cluster node.
Therefore, we have to jump onto the node and talk to kubelet directly:
vagrant ssh kubemaster
sudo su -
# Check if it's running...
systemctl status kubelet
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2022-11-12 10:25:29 UTC; 30s ago
Docs: https://kubernetes.io/docs/home/
Main PID: 29501 (kubelet)
Tasks: 14 (limit: 2339)
Memory: 34.7M
CGroup: /system.slice/kubelet.service
└─29501 /usr/bin/kubelet --bootstrap-kubeconfig=... --kubeconfig=...
To create checkpoint we also need a running Pod. Rather than using system Pods in kube-system, let's create a dummy Nginx webserver in default namespace:
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
kubectl run webserver --image=nginx -n default
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
webserver 1/1 Running 0 27s 10.85.0.4 kubemaster
Above you can see that we also removed taints from the node - this allows us to schedule workloads on the node even though it's part of control-plane.
Next, let's make a sample API request to kubelet to see if we can get any valid response:
curl -skv -X GET "https://localhost:10250/pods" \
--key /etc/kubernetes/pki/apiserver-kubelet-client.key \
--cacert /etc/kubernetes/pki/ca.crt \
--cert /etc/kubernetes/pki/apiserver-kubelet-client.crt
{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {},
"items": [
{
"metadata": {
"name": "webserver",
"namespace": "default",
...
}
}
...
}
kubelet by default runs on port 10250, so we curl it and ask for all its Pods. We also had to specify CA certificate, client certificate and key for authentication.
Now it's time to finally create a checkpoint:
curl -sk -X POST "https://localhost:10250/checkpoint/default/webserver/webserver" \
--key /etc/kubernetes/pki/apiserver-kubelet-client.key \
--cacert /etc/kubernetes/pki/ca.crt \
--cert /etc/kubernetes/pki/apiserver-kubelet-client.crt
# Response:
# {"items":["/var/lib/kubelet/checkpoints/checkpoint-webserver_default-webserver-2022-11-12T10:28:13Z.tar"]}
# Check the directory:
ls -l /var/lib/kubelet/checkpoints/
total 3840
-rw------- 1 root root 3931136 Nov 12 10:28 checkpoint-webserver_default-webserver-2022-11-12T10:28:13Z.tar
# Verify that original container is still running:
crictl ps --name webserver
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
880ee7ddff7f3 docker.io/library/nginx@sha256:... 48 seconds ago Running webserver 0 d584446dd8d5e webserver
The checkpointing API is available at .../checkpoint/${NAMESPACE}/${POD}/${CONTAINER}, here we used the webserver Pod created earlier. This request created an archive in /var/lib/kubelet/checkpoints/checkpoint-<pod>_<namespace>-<container>-<timestamp>.tar.
Depending on the setup you're using, after running the above curl, you might receive error along the lines:
checkpointing of default/webserver/webserver failed (CheckpointContainer is only supported in the CRI v1 runtime API)
# or
checkpointing of default/webserver/webserver failed (rpc error: code = Unknown desc = checkpoint/restore support not available)
That means that your container runtime doesn't (yet) support checkpointing, or it's not enabled correctly.
Analyzing
We now have a checkpointed container archive, so let's take a look at what's inside:
cd /var/lib/kubelet/checkpoints/
# Rename because "tar" doesn't like ":" in names
mv "checkpoint-webserver_default-webserver-2022-11-12T10:28:13Z.tar" webserver.tar
# View contents:
tar --exclude="*/*" -tf webserver.tar
dump.log
checkpoint/
config.dump
spec.dump
rootfs-diff.tar
io.kubernetes.cri-o.LogPath
# Extract:
tar -xf checkpoint-webserver_default-webserver-2022-09-04T10:15:37Z.tar
ls checkpoint/
cgroup.img fdinfo-4.img ids-31.img mountpoints-13.img pages-2.img tmpfs-dev-139.tar.gz.img
core-1.img files.img inventory.img netns-10.img pages-3.img tmpfs-dev-140.tar.gz.img
core-30.img fs-1.img ipcns-var-11.img pagemap-1.img pages-4.img tmpfs-dev-141.tar.gz.img
core-31.img fs-30.img memfd.img pagemap-30.img pstree.img tmpfs-dev-142.tar.gz.img
descriptors.json fs-31.img mm-1.img pagemap-31.img seccomp.img utsns-12.img
fdinfo-2.img ids-1.img mm-30.img pagemap-shmem-94060.img timens-0.img
fdinfo-3.img ids-30.img mm-31.img pages-1.img tmpfs-dev-136.tar.gz.img
cat config.dump
{
"id": "880ee7ddff7f3ce11ee891bd89f8a7356c97b23eb44e0f4fbb51cb7b94ead540",
"name": "k8s_webserver_webserver_default_91ad1757-424e-4195-9f73-349b332cbb7a_0",
"rootfsImageName": "docker.io/library/nginx:latest",
"runtime": "runc",
"createdTime": "2022-11-12T10:27:56.460946241Z"
}
tar -tf rootfs-diff.tar
var/cache/nginx/proxy_temp/
var/cache/nginx/scgi_temp/
var/cache/nginx/uwsgi_temp/
var/cache/nginx/client_temp/
var/cache/nginx/fastcgi_temp/
etc/mtab
run/nginx.pid
run/secrets/kubernetes.io/
run/secrets/kubernetes.io/serviceaccount/
If you don't need a running Pod/container for analysis, then extracting and reading through some of the above shown files might give you the necessary information.
With that said, I'm no security expert, so I'm not going to feed you questionable information here about how to analyze these files.
As a starting point though, you might want check out tools like docker-explorer or this talk on container forensics in Kubernetes.
Restoring
While the Checkpointing API is currently aimed more at forensic analysis, it still can be used to restore the Pod/container from the archive.
The simplest way is to create an image from checkpoint archive:
FROM scratch
# Need to use ADD because it extracts archives
ADD webserver.tar .
Here we use an empty (scratch) image to which we add the archive. We need to use ADD because it automatically extracts archives. Next, we build it with docker or buildah:
cd /var/lib/kubelet/checkpoints/
# Or docker build ...
buildah bud \
--annotation=io.kubernetes.cri-o.annotations.checkpoint.name=webserver \
-t restore-webserver:latest \
Dockerfile .
buildah push localhost/restore-webserver:latest docker.io/martinheinz/restore-webserver:latest
Above we also specify annotation which describes the original human-readable name of the container and then we push to some registry so that Kubernetes can pull it.
Finally, we create a Pod, specifying the previously pushed image:
# pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: restore-webserver
labels:
app: nginx
spec:
containers:
- name: webserver
image: docker.io/martinheinz/restore-webserver:latest
nodeName: kubemaster
To test if it worked, we can expose the Pod via Service and curl its IP:
kubectl expose pod restore-webserver --port=80 --target-port=80
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 14m
restore-webserver ClusterIP 10.104.30.90 <none> 80/TCP 17s
curl http://10.104.30.90
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
</html>
And it worked! We successfully backed-up a running Pod without stopping it and recreated it to its original state.
Closing Thoughts
General checkpoint-and-restore for containers has been possible for a while thanks to CRIU, but this is still a big step for Kubernetes, and hopefully, we will see this feature/API graduate to Beta and eventually GA at some point.
The previous sections demonstrated the usage of checkpointing API - it's very much usable, but it also lacks some basic features, such as native restore functionality or support from all major container runtimes. So, be aware of its limitations if you decide to enable it in production (or even development) environments/clusters.
With that said, this feature is very cool addition not just for forensic analysis - in the future when better/native restore process is available, this could become a proper backup-and-restore process for container workloads, which might be very useful for certain types of long-running Kubernetes workloads.