Kubeeye - Tool To Find Various Problems On Kubernetes, Such As Application Misconfiguration, Unhealthy Cluster Components And Node Problems

KubeEye is an inspection tool for Kubernetes to discover Kubernetes resources (by OPA ), cluster components, cluster nodes (by Node-Problem-Detector) and other configurations are meeting with best practices, and giving suggestions for modification.

KubeEye supports custom inspection rules and plugins installation. Through KubeEye Operator, you can view the inspection results and modification suggestions by the graphical display on the web page.

Architecture

KubeEye get cluster resource details by the Kubernetes API, inspect the resource configurations by inspection rules and plugins, and generate inspection results. See Architecture for details.

How to use

Install KubeEye on your machine
- Download pre built executables from Releases.
- Or you can build from source code
Note: make install will create kubeeye in /usr/local/bin/ on your machine.
```
git clone https://github.com/kubesphere/kubeeye.git
cd kubeeye
make installke
```
[Optional] Install Node-problem-Detector

Note: This will install npd on your cluster, only required if you want detailed report.

kubeeye install npd

Run KubeEye

Note: The results of kubeeye sort by resource kind.

kubeeye audit
KIND          NAMESPACE        NAME                                                           REASON                                        LEVEL    MESSAGE
Node                           docker-desktop                                                 kubelet has no sufficient memory available   warning    KubeletHasNoSufficientMemory
Node                           docker-desktop                                                 kubelet has no sufficient PID available      warning    KubeletHasNoSufficientPID
Node                           docker-desktop                                                 kubelet has disk pressure                    warning    KubeletHasDiskPressure
Deployment    default          testkubeeye                                                                                                                  NoCPULimits
Deployment    default          testkubeeye                                                                                                                  NoReadinessProbe
Deployment    default          testkubeeye                                                                                                                  NotRunAsNonRoot
Deployment    kube-system      coredns                                                                                                               NoCPULimits
Deployment    kube-system      coredns                                                                                                               ImagePullPolicyNotAlways
Deployment    kube-system      coredns                                                                                                               NotRunAsNonRoot
Deployment    kubeeye-system   kubeeye-controller-manager                                                                                            ImagePullPolicyNotAlways
Deployment    kubeeye-system   kubeeye-controller-manager                                                                                            NotRunAsNonRoot
DaemonSet     kube-system      kube-proxy                                                                                                            NoCPULimits
DaemonSet     k          ube-system      kube-proxy                                                                                                            NotRunAsNonRoot
Event         kube-system      coredns-558bd4d5db-c26j8.16d5fa3ddf56675f                      Unhealthy                                    warning   Readiness probe failed: Get "http://10.1.0.87:8181/ready": dial tcp 10.1.0.87:8181: connect: connection refused
Event         kube-system      coredns-558bd4d5db-c26j8.16d5fa3fbdc834c9                      Unhealthy                                    warning   Readiness probe failed: HTTP probe failed with statuscode: 503
Event         kube-system      vpnkit-controller.16d5ac2b2b4fa1eb                             BackOff                                      warning   Back-off restarting failed container
Event         kube-system      vpnkit-controller.16d5fa44d0502641                             BackOff                                      warning   Back-off restarting failed container
Event         kubeeye-system   kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f5fc3229c   Failed                                       warning   Failed to pull image "controller:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for controller, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Event         kubeeye-system   kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f61b28527   Failed                                       warning   Error: ImagePullBackOff
Role          kubeeye-system   kubeeye-leader-election-role                                                                                          CanDeleteResources
ClusterRole                    kubeeye-manager-role                                                                                                  CanDeleteResources
ClusterRole                    kubeeye-manager-role                                                                                                  CanModifyWorkloads
ClusterRole                    vpnkit-controller                                                                                                     CanImpersonateUser
ClusterRole                    vpnkit-controller                                                                                           CanDeleteResources

What KubeEye can do

KubeEye inspects cluster resources according with Kubernetes best practices, to make cluster stable.
KubeEye can find problems of your cluster control plane, including kube-apiserver/kube-controller-manager/etcd, etc.
KubeEye helps you detect all kinds of cluster nodes problems, including memory/cpu/disk pressure, unexpected kernel error logs, etc.

Checklist

YES/NO	CHECK ITEM	Description	Level
✅	PrivilegeEscalationAllowed	Privilege escalation is allowed	danger
✅	CanImpersonateUser	The role/clusterrole can impersonate other user	warning
✅	CanModifyResources	The role/clusterrole can delete kubernetes resources	warning
✅	CanModifyWorkloads	The role/clusterrole can modify kubernetes workloads	warning
✅	NoCPULimits	The resource does not set limits of CPU in containers.resources	danger
✅	NoCPURequests	The resource does not set requests of CPU in containers.resources	danger
✅	HighRiskCapabilities	Have high-Risk options in capabilities such as ALL/SYS_ADMIN/NET_ADMIN	danger
✅	HostIPCAllowed	HostIPC Set to true	danger
✅	HostNetworkAllowed	HostNetwork Set to true	danger
✅	HostPIDAllowed	HostPID Set to true	danger
✅	HostPortAllowed	HostPort Set to true	danger
✅	ImagePullPolicyNotAlways	Image pull policy not always	warning
✅	ImageTagIsLatest	The image tag is latest	warning
✅	ImageTagMiss	The image tag do not declare	danger
✅	InsecureCapabilities	Have insecure options in capabilities such as KILL/SYS_CHROOT/CHOWN	danger
✅	NoLivenessProbe	The resource does not set livenessProbe	warning
✅	NoMemoryLimits	The resource does not set limits of memory in containers.resources	danger
✅	NoMemoryRequests	The resource does not set requests of memory in containers.resources	danger
✅	NoPriorityClassName	The resource does not set priorityClassName	ignore
✅	PrivilegedAllowed	Running a pod in a privileged mode means that the pod can access the host’s resources and kernel capabilities	danger
✅	NoReadinessProbe	The resource does not set readinessProbe	warning
✅	NotReadOnlyRootFilesystem	The resource does not set readOnlyRootFilesystem to true	warning
✅	NotRunAsNonRoot	The resource does not set runAsNonRoot to true, maybe executed run as a root account	warning
✅	CertificateExpiredPeriod	Certificate expiration date less than 30 days	danger
✅	EventAudit	Event audit	warning
✅	NodeStatus	node status audit	warning
✅	DockerStatus	docker status audit	warning
✅	KubeletStatus	kubelet status audit	warning

Add your own inspection rules

Add custom OPA rules

create a directory for OPA rules

mkdir opa

Add custom OPA rules files

Note: the OPA rule for workloads, package name must be kubeeye_workloads_rego for RBAC, package name must be kubeeye_RBAC_rego for nodes, package name must be kubeeye_nodes_rego

Save the following rules to rule file such as imageRegistryRule.rego to check the image registry address complies with rules.

package kubeeye_workloads_rego

deny[msg] {
    resource := input
    type := resource.Object.kind
    resourcename := resource.Object.metadata.name
    resourcenamespace := resource.Object.metadata.namespace
    workloadsType := {"Deployment","ReplicaSet","DaemonSet","StatefulSet","Job"}
    workloadsType[type]

    not workloadsImageRegistryRule(resource)

    msg := {
        "Name": sprintf("%v", [resourcename]),
        "Namespace": sprintf("%v", [resourcenamespace]),
        "Type": sprintf("%v", [type]),
        "Message": "ImageRegistryNotmyregistry"
    }
}

workloadsImageRegistryRule(resource) {
    regex.match("^myregistry.public.kubesphere/basic/.+", resource.Object.spec.template.spec.containers[_].image)
}

Run KubeEye with custom rules

Note: Specify the path then Kubeeye will read all files in the directory that end with .rego.

root:# kubeeye audit -p ./opa
NAMESPACE     NAME              KIND          MESSAGE
default       nginx1            Deployment    [ImageRegistryNotmyregistry NotReadOnlyRootFilesystem NotRunAsNonRoot]
default       nginx11           Deployment    [ImageRegistryNotmyregistry PrivilegeEscalationAllowed HighRiskCapabilities HostIPCAllowed HostPortAllowed ImagePullPolicyNotAlways ImageTagIsLatest InsecureCapabilities NoPriorityClassName PrivilegedAllowed NotReadOnlyRootFilesystem NotRunAsNonRoot]
default       nginx111          Deployment    [ImageRegistryNotmyregistry NoCPULimits NoCPURequests ImageTagMiss NoLivenessProbe NoMemoryLimits NoMemoryRequests NoPriorityClassName NotReadOnlyRootFilesystem NoReadinessProbe NotRunAsNonRoot]

Add custom NPD rules

edit configmap

kubectl edit ConfigMap node-problem-detector-config -n kube-system

restart NPD deployment

kubectl rollout restart DaemonSet node-problem-detector -n kube-system

KubeEye Operator

What is KubeEye Operator

KubeEye Operator is an inspection platform for Kubernetes, manage KubeEye by operator and generate inspection result.

What KubeEye Operator can do

KubeEye Operator provides management functions through web page.
KubeEye Operator recode inspection results by CR, can view and compare cluster inspection results by web page.
KubeEye Operator provides more plugins.
KubeEye Operator provides more detailed modification suggestions.

deploy Kubeeye

kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye.yaml
kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye_insights.yaml

get the inspection results

kubectl get clusterinsight -o yaml

apiVersion: v1
items:
- apiVersion: kubeeye.kubesphere.io/v1alpha1
  kind: ClusterInsight
  metadata:
    name: clusterinsight-sample
    namespace: default
  spec:
    auditPeriod: 24h
  status:
    auditResults:
      auditResults:
      - resourcesType: Node
        resultInfos:
        - namespace: ""
          resourceInfos:
          - items:
            - level: warning
              message: KubeletHasNoSufficientMemory
              reason: kubelet has no sufficient memory available
            - level: warning
              message: KubeletHasNoSufficientPID
              reason: kubelet has no sufficient PID available
            - level: warning
              message: KubeletHasDiskPressure
              reason: kubelet has disk pressure
            name: kubeeyeNode