Introduction to probes in Kubernetes

Health Check Introduction:

The initiator performs periodic health status detection of the container. Container 1: Periodic testing is equivalent to periodic physical examination for humans, and each test is equivalent to the content of each physical examination for humans

Docker Health Check implementation:

1. Implement probes in docker-compose:

root@k8s-master:~/yaml/1226/2.PodProbe-case/case2-docker-container-healthy-check# cat 1.docker-compose.yaml
version: '3.6'
services:
  nginx-service:
    image: nginx:1.20.2
    container_name: nginx-web1
    expose:
      - 80
      - 443
    ports:
      - "80:80"
      - "443:443"
    restart: always
    healthcheck: #添加服务健康状态检查
      test: ["CMD", "curl", "-f", "http://localhost"]
      interval: 5s #健康状态检查的间隔时间，默认为30s
      timeout: 5s #单次检查的失败超时时间，默认为30s
      retries: 3 #连续失败次数默认3次，当连续失败retries次数后将容器置为unhealthy状态
      start_period: 60s #60s后每间隔interval的时间检查一次，连续retries次后才将容器置为unhealthy状态, 但是start_period时间内检查成功就认为是检查成功并装容器置于healthy状态           

2. Implement health status detection in dockerfile:

root@k8s-master:~/yaml/1226/2.PodProbe-case/case2-docker-container-healthy-check# cat 2.Dockerfile
FROM nginx:1.20.2

maintainer "[email protected]"


HEALTHCHECK --interval=5s --timeout=2s --retries=3 \
  CMD curl --silent --fail localhost:80 || exit 1           

• The container is in the starting state until the detection passes
• After the detection passes (the detection returns a status code of 0), the healthy state is displayed
• Failed detection (detection return status code is 1) followed by unhealthy status (self-parameter test)

Kubernetes pod lifecycle

The life cycle of the pod, postStart detection can be configured from start, livenessProbe and readinessProbe can be configured during operation, and finally preStop operation can be configured before stop.

Pod restart policy:

Once the pod is configured with probes, when the detection fails, the next step will be performed on the pod based on the restartPolicy:

restartPolicy:

• Always: When the container is abnormal, k8s automatically restarts the container, ReplicationController/Replicaset/Deployment, the default is Always.
• OnFailure: When the container fails (the container stops running and the exit code is not 0), k8s automatically restarts the container.
• Never: The container, Job, or CronJob is not restarted regardless of the state of the container.

imagePullPolicy:

• IfNotPresent: If the node node does not have this image, it will go to the specified image repository to pull it, and the node node will use the node local image if it does.
• Always: Each time the pod is rebuilt, the image will be re-pulled
• Never: Images are pulled from the image center and only local images are used

Third, the probe

3.1 Probe introduction

Probes are periodic diagnostics performed by the kubelet on the container to ensure that the state of the pod is always running, and to perform the diagnosis, the kubelet calls the handler implemented by the container, also known as a hook.

There are three types of handlers:

• ExecAction: Executes the specified command in the container, and if the return code is 0 when the command exits, the diagnosis is considered successful.
• TCPSocketAction: TCP checks the IP address of the container on the specified port, and if the port is open, the diagnosis is considered successful.
• HTTPGetAction: Performs an HTTPGet request on the specified port and the IP address of the container on the path, and if the status code of the response is greater than or equal to 200 and less than 400, the diagnosis is considered successful.

Each probe will result in one of three results:

• Success: The container passed diagnostics.
• Failed: The container failed diagnostics.
• Unknown: The diagnosis failed, so no action was taken.

3.2 Probe types

• startupProbe: Startup probe, kubernetes v1.16 introduces to determine whether the application in the container has been started, if the startup probe is configured, all other probes will be disabled first until the startupProbe detection is successful, if the startupProbe probe fails, the kubelet will kill the container, the container will proceed to the next step according to the restart policy, if the container does not provide a startup probe, the default status is success
• livenessProbe: The liveness probe detects if the container container is running, if the liveness probe fails, the kubelet kills the container and the container will be affected by its restart policy, if the container does not provide a survival probe, the default state is Success, livenessProbe is used to control whether the pod is restarted.
• readinessProbe: If the readiness probe fails, the endpoint controller will remove the pod's IP address from the endpoints of all services that match the pod, the readiness state before the initial delay defaults to Failure, if the container does not provide a readiness probe, the default state is Success, and the readinessProbe is used to control whether the pod is added to the service.

3.3 Probe general configuration parameters:

• initialDelaySeconds: 120 initialization delay time, tells the kubelet how many seconds to wait before performing the first probe, the default is 0 seconds, and the minimum value is 0.
• periodSeconds: 60 probe cycle interval, specifies how many seconds the kubelet should perform a liveness probe, the default is 10 seconds. The minimum value is 1.
• timeoutSeconds: 5 The timeout period of a single probe, how many seconds to wait after the timeout of the probe, the default value is 1 second, the minimum value is 1.
• successThreshold: 1 The number of retries from failure to success, the minimum number of consecutive successes that the probe is considered successful after failure, the default value is 1, this value of the liveness probe must be 1, and the minimum value is 1.
• failureThreshold: 3 The number of retries from success to failure, when the pod is started and detected to fail, the number of retries of Kubernetes, abandonment in the case of survival detection means restarting the container, abandoning the pod in the case of ready detection will be marked as not ready, the default value is 3, the minimum value is 1.

3.4 Probe HTTP configuration parameters

HTTP probes can configure additional fields on httpGet:

• host: The hostname used by the connection, the default is the IP of the pod, or you can set "host" in the HTTP header instead.
• scheme: http is used to set the way to connect to the host (HTTP or HTTPS), the default is HTTP.
• path: /monitor/index.html The path to the HTTP service.
• httpHeaders: Custom HTTP headers in the request, HTTP header fields allowed to be duplicated.
• port: 80 The port number or port name of the access container, if the number must be between 1 ~ 65535.

3.5 Probe example

case3-Probe# kubectl apply -f 5-startupProbe-livenessProbe-readinessProbe.yaml
root@k8s-master:~/yaml/1226/2.PodProbe-case/case3-Probe# cat 5-startupProbe-livenessProbe-readinessProbe.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myserver-myapp-frontend-deployment
  namespace: myserver
spec:
  replicas: 1
  selector:
    matchLabels: #rs or deployment
      app: myserver-myapp-frontend-label
    #matchExpressions:
    #  - {key: app, operator: In, values: [myserver-myapp-frontend,ng-rs-81]}
  template:
    metadata:
      labels:
        app: myserver-myapp-frontend-label
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: myserver-myapp-frontend-label
        image: nginx:1.20.2
        ports:
        - containerPort: 80
        startupProbe:
          httpGet:
            #path: /monitor/index.html
            path: /index.html
            port: 80
          initialDelaySeconds: 5 #首次检测延迟5s
          failureThreshold: 3  #从成功转为失败的次数
          periodSeconds: 3 #探测间隔周期
        readinessProbe:
          httpGet:
            #path: /monitor/monitor.html
            path: /index.html
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 3
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
        livenessProbe:
          httpGet:
            #path: /monitor/monitor.html
            path: /index.html
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 3
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3

---
apiVersion: v1
kind: Service
metadata:
  name: myserver-myapp-frontend-service
  namespace: myserver
spec:
  ports:
  - name: http
    port: 81
    targetPort: 80
    nodePort: 30012
    protocol: TCP
  type: NodePort
  selector:
    app: myserver-myapp-frontend-label
           

Debug to see how the pods are performing

3.6 Introduction to postStart and preStop

PostStart and preStop handlers:

• postStart: Detects pods as soon as they are created, i.e. does not wait for services in the pod to start. If the postStart execution fails, the pod will not continue to be created.
• preStop: Executes before the pod is stopped

The termination process of the pod

1. Create a pod

• Complete the scheduling process
• The container starts and executes postStart
• livenessProbe enters the running state
• readinessProbe service associated pods
• Accept client requests

2. Delete the pod

• The pod is set to the "Terminating" state
• Removed from the service's list of endpoints and no longer accepts client requests.
• Perform PreStop
• Kubernetes sends a SIGTERM signal to the container in the pod to terminate the main process inside the pod, which lets the container know that it will be shut down soon.
• terminationGracePeriodSeconds: 60 optional termination waiting period, if there is a delete grace time set, wait for the grace time to expire, otherwise wait up to 30s, Kubernetes wait for the specified time is called graceful termination grace period, by default it is 30 seconds, it is worth noting that the waiting period is executed in parallel with the preStop Hook and SIGTERM signals, That is, Kubernetes may not wait for the preStop Hook to complete (forcibly terminate the pod after up to 30 seconds before the main process has finished).
• The SIGKILL signal is sent to the pod and the pod is deleted

Sample code:

case4-postStart-preStop# kubectl apply -f 1-myserver-myapp1-postStart-preStop.yaml
root@k8s-master:~/yaml/1226/2.PodProbe-case/case4-postStart-preStop# cat 1-myserver-myapp1-postStart-preStop.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myserver-myapp1-lifecycle
  labels:
    app: myserver-myapp1-lifecycle
  namespace: myserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myserver-myapp1-lifecycle-label
  template:
    metadata:
      labels:
        app: myserver-myapp1-lifecycle-label
    spec:
      terminationGracePeriodSeconds: 60 #可选终止等待期，如果有设置删除宽限时间，则等待宽限时间到期，否则最多等待30s，
      containers:
      - name: myserver-myapp1-lifecycle-label
        image: tomcat:7.0.94-alpine
        lifecycle:
          postStart:
            exec:
             #command: 把自己注册到注册在中心
              command: ["/bin/sh", "-c", "echo 'Hello from the postStart handler' >> /usr/local/tomcat/webapps/ROOT/index.html"]

            #httpGet:
            #  #path: /monitor/monitor.html
            #  host: www.magedu.com
            #  port: 80
            #  scheme: HTTP
            #  path: index.html
          preStop:
            exec:
             #command: 把自己从注册中心移除
              command: ["/usr/local/tomcat/bin/catalina.sh","stop"]
        ports:
          - name: http
            containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: myserver-myapp1-lifecycle-service
  namespace: myserver
spec:
  ports:
  - name: http
    port: 80
    targetPort: 8080
    nodePort: 30012
    protocol: TCP
  type: NodePort
  selector:
    app: myserver-myapp1-lifecycle-label