the site subtitle

Containerd gvisor & k8s

2019.11.25

本文主要讨论如何在containerd中使用gvisor作为runtime,以及如何对接k8s。

安装containerd

这个就不用多说了,直接apt install containerd 即可,会安装一下几个可执行文件,如下所示

/usr/bin/containerd 守护进程
/usr/bin/containerd-shim 应该是负责处理容器内stdout & stderr
/usr/bin/containerd-shim-runc-v1 应该是负责启动容器内的pid 1
/usr/bin/containerd-stress 压力测试工具
/usr/bin/ctr cli工具

吐槽一下 ctr可能没有docker的cli那么好用

接下来我们来运行一个容器测试一下containerd:

  • 因为containerd默认的pause容器使用的是gcr.k8s.io/pause,如果网络有问题的话,就需要提前导入,或者在修改默认配置。
  • 还要提醒一下,containerd有namespace的区别,默认是在default下面,这里不是k8s的namespace 。
  • 除此之外还要配置CNI以供容器启动分配网络,可以使用下面的配置:
mkdir -p /etc/cni/net.d
cat >'/etc/cni/net.d'/10-containerd-net.conflist <<EOF
{
  "cniVersion": "0.3.1",
  "name": "containerd-net",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni1",
      "isGateway": true,
      "ipMasq": true,
      "promiscMode": true,
      "ipam": {
        "type": "host-local",
        "subnet": "10.88.0.0/16",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ]
      }
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true}
    }
  ]
}
EOF

接下来就可以真正开始运行容器了。

先拉镜像
ctr i pull docker.io/library/nginx:alpine 
运行
ctr run -d docker.io/library/nginx:alpine demo
检查一下 containerd会把容器作为task
ctr c ls
CONTAINER    IMAGE                             RUNTIME
demo         docker.io/library/nginx:alpine    io.containerd.runtime.v1.linux
 ctr t ls
TASK    PID      STATUS
demo    19901    RUNNING

这里停掉containerd的守护进程,可以发现已经运行的容器是不受影响的
pstree
systemd─┬─VBoxService───7*[{VBoxService}]
        ├─accounts-daemon───2*[{accounts-daemon}]
        ├─2*[agetty]
        ├─atd
        ├─containerd-shim─┬─nginx───2*[nginx]
        │                 └─9*[{containerd-shim}]
        ├─cron
        ├─dbus-daemon
        ├─irqbalance───{irqbalance}
        ├─lvmetad
        ├─lxcfs───6*[{lxcfs}]
        ├─networkd-dispat───{networkd-dispat}
        ├─polkitd───2*[{polkitd}]
        ├─rsyslogd───3*[{rsyslogd}]
        ├─sshd───sshd───sshd───bash───sudo───su───bash─┬─containerd───7*[{containerd}]
        │                                              └─pstree
        ├─systemd───(sd-pam)
        ├─systemd-journal
        ├─systemd-logind
        ├─systemd-network
        ├─systemd-resolve
        ├─systemd-udevd
        └─unattended-upgr───{unattended-upgr}

配置gvisor

首先需要编译或者下载runsc,也就是gvisor的运行时,然后需要下载containerd-shim-runsc-v1 放到PATH任意一个目录中

# 示例配置
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    enable_selinux = false
    sandbox_image = "gcr.azk8s.cn/google_containers/pause:3.1"
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      no_pivot = false
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
      [plugins.cri.containerd.runtimes.runsc]
        runtime_type = "io.containerd.runsc.v1"
    [plugins.cri.cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
    [plugins.cri.x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins.opt]
    path = "/opt/containerd"
  [plugins.restart]
    interval = "10s"
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"

对接k8s

对于kubeadm,初始化的时候指定containerd的sock即可,如下所示:

kubeadm init --cri-socket /run/containerd/containerd.sock --image-repository gcr.azk8s.cn/google_containers  --kubernetes-version 1.16.3  --apiserver-advertise-address 192.168.2.2 --pod-network-cidr=10.244.0.0/16 

# 创建runtimeclass
cat <<EOF | kubectl apply -f -
apiVersion: node.k8s.io/v1beta1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
EOF

# 运行pod
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx-gvisor
spec:
  runtimeClassName: gvisor
  containers:
  - name: nginx
    image: nginx:alpine
EOF

Ref

Doc

https://github.com/google/gvisor-containerd-shim/blob/master/docs/runtime-handler-shim-v2-quickstart.md