Life is too bug

Kubernetes 认证与授权

2018.06.15

对于一个系统来说,认证和授权不可或缺的部分,最常见的方式要数账号密码,管理员和普通用户这种认证方式,也就是常说的ABAC。但是每次修改权限都需要重启,或者修改记录,会有些不方便,于是基于角色的访问控制--RBAC就出现了,同时k8s分离了认证,授权和准入的过程,方便第三方开发相关插件。

下图大致描述了k8s认证的流程图,描述了一个请求在经过APIServer的时候都经历了哪些鉴权的过程。如下图所示:需要访问API的有人类,也就是Ops,通过kubectl以及Kubeconfig配置文件和API-Server交互。还有就是Pod,也就是应用,对API资源进行CURD。然后依次经过了身份认证(authentication)、授权(authorization)和准入控制(admission control)。下面就来详细介绍下认证和授权

认证

  • Authentication:即身份验证,这个环节它面对的输入是整个http request,它负责对来自client的请求进行身份校验,支持的方法包括:
    • client证书验证(https双向验证)
    • basic auth
    • 普通token
    • jwt token(用于serviceaccount)

APIServer启动时,可以指定一种Authentication方法,也可以指定多种方法。如果指定了多种方法,那么APIServer将会逐个使用这些方法对客户端请求进行验证,只要请求数据通过其中一种方法的验证,APIServer就会认为Authentication成功;在较新版本kubeadm引导启动的k8s集群的apiserver初始配置中,默认支持client证书验证和serviceaccount两种身份验证方式。在这个环节,apiserver会通过client证书或http header中的字段(比如serviceaccount的jwt token)来识别出请求的“用户身份”,包括”user”、”group”等,这些信息将在后面的authorization环节用到。

Client 证书

首先我们看一下kubectl的默认配置文件~/.kube/config

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: https://172.16.66.101:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

这有两个重要的信息 client-certificate-data, client-key-data,首先把data提取出来

# 生成client-certificate-data
grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt

# 生成client-key-data
grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key

openssl x509 -noout -text -in kubecfg.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6675913985209031083 (0x5ca59c66277259ab)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Apr 13 10:22:37 2019 GMT
            Not After : Apr 12 10:22:38 2020 GMT
        Subject: O=system:masters, CN=kubernetes-admin #此证书是签发给system:masters的,
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    .....
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication # 此证书可以作为TLS的客户端证书
    Signature Algorithm: sha256WithRSAEncryption

为什么是这个用户呢?在集群中可以看到这样一条记录,k8s默认把system:masters用户绑定上了clsuter-admin的角色,也就是整个集群的管理员,所以当我们有了这个客户端证书之后,就可以管理集群的任何资源。

kubectl -n kube-system describe clusterrolebindings cluster-admin

Name:         cluster-admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate=true
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind   Name            Namespace

下面我们来手动访问下API Server:

MASTER=192.168.1.1
# 因为是自签名证书 用-k 忽略证书校验 
# 默认的request是system:anonymous用户 没有权限访问
curl -k  https://$MASTER:6443/api
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/api\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}
# 使用客户端证书进行认证
curl -v --cacert /etc/kubernetes/pki/ca.crt  --key ./kubecfg.key --cert ./kubecfg.crt https://$MASTER:6443/api
# 同理访问kubelet的metrics接口
curl -k  --cacert /etc/kubernetes/pki/ca.crt --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key https://127.0.0.1:10250/metrics

# 也可以生成另一类p12证书
openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12 -name "kubernetes-client"

Basic Auth

API有一个启动参数--basic-auth-file,可以指定一个csv格式的文件作为Basic Auth

cat <<EOF > basicauth.csv
admin,admin,1,"demo"
EOF

cat <<EOF > kubectl apply -f -
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ui-admin
rules:
- apiGroups:
  - ""
  resources:
  - services
  - services/proxy
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ui-admin-binding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ui-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: admin
EOF  

Service Account

当创建一个Service Account会生成一个token,通过kubectl -n kube-system describe secrets ${Service account}可以看到一个以eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9开头的Token。可以在jwt.io上面解码一下,得到的结果如下所示:APIServer也就是通过这些字段进行鉴权的。

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "default",
  "kubernetes.io/serviceaccount/secret.name": "default-token-ph2f5",
  "kubernetes.io/serviceaccount/service-account.name": "default",
  "kubernetes.io/serviceaccount/service-account.uid": "7ce56043-7dcc-11e9-9925-00163e132347",
  "sub": "system:serviceaccount:default:default"
}
# 手动使用Token认证一下
TOKEN=$(kubectl describe secret $(kubectl get secrets | grep default | cut -f1 -d ' ') | grep -E '^token' | cut -f2 -d':' | tr -d '\t')
MASTER=10.96.0.1
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -k -H "Authorization: Bearer $TOKEN" https://$MASTER:6443/api

授权

Authorization:授权。这个环节面对的输入是http request context中的各种属性,包括:user、group、request path(比如:/api/v1、/healthz、/version等)、request verb(比如:get、list、create等)。APIServer会将这些属性值与事先配置好的访问策略(access policy)相比较。APIServer支持多种authorization mode,包括Node、RBAC、Webhook等。

Authorization flags:

      --authorization-mode strings
                Ordered list of plug-ins to do authorization on secure port. Comma-delimited list of: AlwaysAllow,AlwaysDeny,ABAC,Webhook,RBAC,Node. (default [AlwaysAllow])
      --authorization-policy-file string
                File with authorization policy in csv format, used with --authorization-mode=ABAC, on the secure port.
      --authorization-webhook-cache-authorized-ttl duration
                The duration to cache 'authorized' responses from the webhook authorizer. (default 5m0s)
      --authorization-webhook-cache-unauthorized-ttl duration
                The duration to cache 'unauthorized' responses from the webhook authorizer. (default 30s)
      --authorization-webhook-config-file string
                File with webhook configuration in kubeconfig format, used with --authorization-mode=Webhook. The API server will query the remote service to determine access on the API server's secure port.

APIServer启动时,可以指定一种authorization mode,也可以指定多种authorization mode,如果是后者,只要Request通过了其中一种mode的授权,那么该环节的最终结果就是授权成功。在较新版本kubeadm引导启动的k8s集群的apiserver初始配置中,authorization-mode的默认配置是”Node,RBAC”。Node授权器主要用于各个node上的kubelet访问apiserver时使用的,其他一般均由RBAC授权器来授权。

准入

Admission flags:

      --admission-control strings
                Admission is divided into two phases. In the first phase, only mutating admission plugins run. In the second phase, only validating admission plugins run. The names in the below list may
                represent a validating plugin, a mutating plugin, or both. The order of plugins in which they are passed to this flag does not matter. Comma-delimited list of: AlwaysAdmit, AlwaysDeny,
                AlwaysPullImages, DefaultStorageClass, DefaultTolerationSeconds, DenyEscalatingExec, DenyExecOnPrivileged, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, Initializers,
                LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction,
                OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodPreset, PodSecurityPolicy, PodTolerationRestriction, Priority,
                ResourceQuota, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. (DEPRECATED: Use --enable-admission-plugins or
                --disable-admission-plugins instead. Will be removed in a future version.)
      --admission-control-config-file string
                File with admission control configuration.
      --disable-admission-plugins strings
                admission plugins that should be disabled although they are in the default enabled plugins list (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, Priority,
                DefaultTolerationSeconds, DefaultStorageClass, PersistentVolumeClaimResize, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission
                plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, DefaultStorageClass, DefaultTolerationSeconds, DenyEscalatingExec, DenyExecOnPrivileged, EventRateLimit, ExtendedResourceToleration,
                ImagePolicyWebhook, Initializers, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction,
                OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodPreset, PodSecurityPolicy, PodTolerationRestriction, Priority,
                ResourceQuota, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
      --enable-admission-plugins strings
                admission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, Priority, DefaultTolerationSeconds,
                DefaultStorageClass, PersistentVolumeClaimResize, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit,
                AlwaysDeny, AlwaysPullImages, DefaultStorageClass, DefaultTolerationSeconds, DenyEscalatingExec, DenyExecOnPrivileged, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook,
                Initializers, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction,
                OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodPreset, PodSecurityPolicy, PodTolerationRestriction, Priority,
                ResourceQuota, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.

经过了前面两个环境之后,就到了admission环节。API Server 可以通过-enable-admission-plugins参数指定加载的授权插件,目前有以下插件

  • AlwaysAdmit:允许所有请求通过
  • AlwaysPullImages:在启动容器之前总是去下载镜像,相当于每当容器启动前做一次用于是否有权使用该容器镜像的检查
  • AlwaysDeny:禁止所有请求通过,用于测试
  • DenyEscalatingExec:拒绝exec和attach命令到有升级特权的Pod的终端用户访问。如果集中包含升级特权的容器,而要限制终端用户在这些容器中执行命令的能力,推荐使用此插件
  • ImagePolicyWebhook
  • ServiceAccount:这个插件实现了serviceAccounts等等自动化,如果使用ServiceAccount对象,强烈推荐使用这个插件
  • SecurityContextDeny:将Pod定义中定义了的SecurityContext选项全部失效。SecurityContext包含在容器中定义了操作系统级别的安全选型如fsGroup,selinux等选项
  • ResourceQuota:用于namespace上的配额管理,它会观察进入的请求,确保在namespace上的配额不超标。推荐将这个插件放到准入控制器列表的最后一个。ResourceQuota准入控制器既可以限制某个namespace中创建资源的数量,又可以限制某个namespace中被Pod请求的资源总量。ResourceQuota准入控制器和ResourceQuota资源对象一起可以实现资源配额管理。
  • LimitRanger:用于Pod和容器上的配额管理,它会观察进入的请求,确保Pod和容器上的配额不会超标。准入控制器LimitRanger和资源对象LimitRange一起实现资源限制管理
  • NamespaceLifecycle:当一个请求是在一个不存在的namespace下创建资源对象时,该请求会被拒绝。当删除一个namespace时,将会删除该namespace下的所有资源对象

RBAC实战

k8s 大约有以下几种角色

  • role 和 clusterrole 规定了权限的内容,也就是某个人或者角色能干什么
  • Service account 也就是具体的角色或者账户
  • rolebinding 和 clusterrolebinding 就是将两者结合在一起,规定什么人有什么样的权限

下面我们来看一下例子

---
# ------------------- ClusterRole ------------------- #
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
# ------------------- ClusterRoleBinding ------------------- #
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system
---
# ------------------- ServiceAccount ------------------- #
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system

审计

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Request",
  "auditID": "43af9fa0-d080-4f7b-bcf4-31fe4504348e",
  "stage": "ResponseComplete",
  "requestURI": "/apis/policy/v1beta1/podsecuritypolicies?resourceVersion=1&timeout=9m58s&timeoutSeconds=598&watch=true",
  "verb": "watch",
  "user": {
    "username": "system:kube-controller-manager",
    "groups": [
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "192.168.1.11"
  ],
  "userAgent": "kube-controller-manager/v1.13.1 (linux/amd64) kubernetes/eec55b9/shared-informers",
  "objectRef": {
    "resource": "podsecuritypolicies",
    "apiGroup": "policy",
    "apiVersion": "v1beta1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "requestReceivedTimestamp": "2019-03-28T09:52:43.369727Z",
  "stageTimestamp": "2019-03-28T09:59:16.037365Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""
  }
}

Ref

Kubernetes集群的安全配置

https://tonybai.com/2016/11/25/the-security-settings-for-kubernetes-cluster/

Kubernetes RBAC 详解 https://blog.qikqiak.com/post/use-rbac-in-k8s/

使用 RBAC 控制 kubectl 权限

https://mritd.me/2018/03/20/use-rbac-to-control-kubectl-permissions/

comments powered by Disqus