kubernetes hpa(Horizo​​ntal Pod Autoscaler)示例(48)

应用运维 2023-07-16 Escape 手机阅读
Scan me!

hpa

hpa有着非常强大的功能，支持应用规模的自动伸缩。如果某一个pod，资源利用率达到一个临界值后，会自动的修改Deployment的replicas值，这种方式也有自动计算结果得到的。比如说，我有5个pod，我期望每个负载率是90% 就进行扩展一个pod，依次类推。当负载降下来的时候就在自动所容到正常的个数。

20181212224102.png

hpa是有两个版本，v1使用核心指标定义，而核心指标中只有cpu和内存，而内存不可压缩资源，也不支持弹性压缩，那就只能使用cpu指标进行伸缩

[root@linuxea linuxea]# kubectl api-versions
autoscaling/v1
autoscaling/v2beta1

在explain hpa.spec中：maxReplicas：最大副本数minReplicas：最少副本数targetCPUUtilizationPercentage：cpu评估，cpu利用率到多少就开始伸缩

在hpa.spec.scaleTargetRef： 基于资源伸缩标准

资源自动伸缩

我们创建一个pod，并且做资源限制.yaml如下：

[root@linuxea linuxea]# cat deploy-hpa.yaml 
apiVersion: apps/v1
kind: Deployment
metadata: 
  name: linuxea-hpa
  namespace: default
  labels:
    www: linuxea-com
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      version: v0.1.32
  template:
    metadata:
      labels:
        version: v0.1.32  
    spec:               
      containers:
      - name: nginx-hpa
        image: marksugar/nginx:1.14.b
        ports:
        - name: http
          containerPort: 80
        resources:
          requests:
            cpu: "300m"
            memory: "256Mi"
          limits:
            cpu: "1"
            memory: "512Mi"
[root@linuxea linuxea]# kubectl apply -f deploy-hpa.yaml 
deployment.apps/linuxea-hpa created
[root@linuxea linuxea]# kubectl get pods
NAME                           READY     STATUS    RESTARTS   AGE
linuxea-hpa-6f7b8ddb67-cfcdw   1/1       Running   0          5s

而后配置service

[root@linuxea linuxea]# cat deploy-hpa-svc.yaml 
kind: Service
apiVersion: v1
metadata:
  name: linuxea-hpa
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30080
  selector:
    version: v0.1.32
[root@linuxea linuxea]# kubectl apply -f deploy-hpa-svc.yaml 
service/linuxea-hpa created
[root@linuxea linuxea]# kubectl get svc
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP        17h
linuxea-hpa   NodePort    10.105.186.30   <none>        80:30080/TCP   6s

这样一来就可以通过30080访问

autoscale

  • autoscale

我们定义自动伸缩，创建一个autoscale的控制器。kubectl autoscale --help

指明最多有几个，最少有几个，cpu利用率最多不能超过多少

  # Auto scale a deployment "foo", with the number of pods between 2 and 10, no target CPU utilization specified so a
default autoscaling policy will be used:
  kubectl autoscale deployment foo --min=2 --max=10
  
  # Auto scale a replication controller "foo", with the number of pods between 1 and 5, target CPU utilization at 80%:
  kubectl autoscale rc foo --max=5 --cpu-percent=80

使用autoscale定义，最少一个，最多八个，cpu利用率最多不能使用超过60

[root@linuxea linuxea]# kubectl autoscale deployment linuxea-hpa --min=1 --max=8 --cpu-percent=60
horizontalpodautoscaler.autoscaling/linuxea-hpa autoscaled

像这样

[root@linuxea linuxea]# kubectl get hpa
NAME        REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
linuxea-hpa Deployment/linuxea-hpa   <unknown>/60%   1         8         0          4s
[root@linuxea linuxea]# kubectl get hpa
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
linuxea-hpa   Deployment/linuxea-hpa   0%/60%    1         8         1          7s

我们做一些压力测试，如果CPU到60%就会创建新的pod

压力测试

[root@linuxea linuxea]#  ab -c 1000 -n 150099 http://10.10.240.161:30080/linuxea.html
[root@linuxea linuxea]# kubectl describe hpa
Name:                                                  linuxea-hpa
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sun, 11 Nov 2018 07:29:54 +0000
Reference:                                             Deployment/linuxea-hpa
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  105% (316m) / 60%
Min replicas:                                          1
Max replicas:                                          8
Deployment pods:                                       1 current / 2 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 2
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  6s    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target

随着负载的上涨，pod也会创建

[root@linuxea linuxea]# kubectl get pods
NAME                           READY     STATUS    RESTARTS   AGE
linuxea-hpa-6f7b8ddb67-hrb64   1/1       Running   0          10m
linuxea-hpa-6f7b8ddb67-mgnkc   1/1       Running   0          18s

那么现在已经运行了两个

[root@linuxea linuxea]# kubectl get hpa
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
linuxea-hpa   Deployment/linuxea-hpa   0%/60%    1         8         2          11m

一旦负载超过阈值就会创建预设的最大值和最小值

[root@linuxea linuxea]# kubectl get hpa -w
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
linuxea-hpa   Deployment/linuxea-hpa   0%/60%    1         8         2          12m
linuxea-hpa   Deployment/linuxea-hpa   23%/60%   1         8         2         12m
linuxea-hpa   Deployment/linuxea-hpa   0%/60%    1         8         2         13m
linuxea-hpa   Deployment/linuxea-hpa   0%/60%    1         8         2         13m
linuxea-hpa   Deployment/linuxea-hpa   122%/60%   1         8         2         14m
linuxea-hpa   Deployment/linuxea-hpa   144%/60%   1         8         4         14m
[root@linuxea linuxea]# kubectl get pods -w
NAME                           READY     STATUS    RESTARTS   AGE
linuxea-hpa-6f7b8ddb67-hrb64   1/1       Running   0          11m
linuxea-hpa-6f7b8ddb67-mgnkc   1/1       Running   0          1m
 
linuxea-hpa-6f7b8ddb67-7bn99   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-7fl4c   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-7bn99   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-7fl4c   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-7bn99   0/1       ContainerCreating   0         0s
linuxea-hpa-6f7b8ddb67-7fl4c   0/1       ContainerCreating   0         0s
linuxea-hpa-6f7b8ddb67-7bn99   1/1       Running   0         1s
linuxea-hpa-6f7b8ddb67-7fl4c   1/1       Running   0         1s
[root@linuxea ~]# kubectl describe hpa
Name:                                                  linuxea-hpa
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sun, 11 Nov 2018 07:29:54 +0000
Reference:                                             Deployment/linuxea-hpa
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  144% (433m) / 60%
Min replicas:                                          1
Max replicas:                                          8
Deployment pods:                                       4 current / 4 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     False   BackoffBoth         the time since the previous scale is still within both the downscale and upscale forbidden windows
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  4m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  37s   horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target

默认使用的autocale的v1控制器

autoscaling/v2beta1

使用autoscaling/v2beta1，kind为HorizontalPodAutoscaler。对linuxea-hpa进行扩展，如下

spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: linuxea-hpa

对这个资源minReplicas最少一个，maxReplicas最大5个，metrics对cpu资源进行评估，当cpu使用30%就进行自动扩展，内存超过50M（targetAverageValue只能是值）

  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 30
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 50Mi

如下：

[root@linuxea ~]# cat autoscale-hpa.yaml 
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: linuxea-hpa-2
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: linuxea-hpa
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 30
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 50Mi

删掉v1版本的，而后apply v2beta1

[root@linuxea ~]# kubectl delete hpa linuxea-hpa
horizontalpodautoscaler.autoscaling "linuxea-hpa" deleted
[root@linuxea ~]# kubectl apply -f autoscale-hpa.yaml
horizontalpodautoscaler.autoscaling/linuxea-hpa-2 configured

此时就可以对CPU和内存资源做评估

[root@linuxea ~]# kubectl get hpa
NAME            REFERENCE              TARGETS                         MINPODS   MAXPODS   REPLICAS   AGE
linuxea-hpa-2   Deployment/linuxea-hpa <unknown>/50Mi, <unknown>/30%   1   10 5   3s

如果压力满足就扩展，否则就缩减

压测

[root@linuxea linuxea]# ab -c 1000 -n 150099 http://10.10.240.161:30080/linuxea.html
[root@linuxea ~]# kubectl get hpa
NAME            REFERENCE                TARGETS                      MINPODS   MAXPODS   REPLICAS   AGE
linuxea-hpa-2   Deployment/linuxea-hpa   107837030400m/50Mi, 0%/30%   1  10  5   6m

当检测到压测,进行扩展

[root@linuxea linuxea]# kubectl get pods -w
NAME                           READY     STATUS    RESTARTS   AGE
linuxea-hpa-6f7b8ddb67-7bn99   1/1       Running   0          3m
linuxea-hpa-6f7b8ddb67-7fl4c   1/1       Running   0          3m
linuxea-hpa-6f7b8ddb67-c8ssz   1/1       Running   0          39s
linuxea-hpa-6f7b8ddb67-hrb64   1/1       Running   0          18m
linuxea-hpa-6f7b8ddb67-mgnkc   1/1       Running   0          7m
linuxea-hpa-6f7b8ddb67-l89sk   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-vfkqj   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-l89sk   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-f96n5   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-vfkqj   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-gpxkt   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-f96n5   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-5kwtl   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-gpxkt   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-5kwtl   0/1       Pending   0         0s
linuxea-hpa-6f7b8ddb67-vfkqj   0/1       ContainerCreating   0         0s
linuxea-hpa-6f7b8ddb67-l89sk   0/1       ContainerCreating   0         0s
linuxea-hpa-6f7b8ddb67-f96n5   0/1       ContainerCreating   0         0s
linuxea-hpa-6f7b8ddb67-5kwtl   0/1       ContainerCreating   0         0s
linuxea-hpa-6f7b8ddb67-gpxkt   0/1       ContainerCreating   0         0s
linuxea-hpa-6f7b8ddb67-l89sk   1/1       Running   0         1s
linuxea-hpa-6f7b8ddb67-gpxkt   1/1       Running   0         1s
linuxea-hpa-6f7b8ddb67-5kwtl   1/1       Running   0         2s
linuxea-hpa-6f7b8ddb67-f96n5   1/1       Running   0         2s
linuxea-hpa-6f7b8ddb67-vfkqj   1/1       Running   0         2s

可使用kubectl describe hpa 查看

[root@linuxea ~]# kubectl describe hpa 
Name:                                                  linuxea-hpa-2
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"linuxea-hpa-2","namespace":"default"},"spec":...
CreationTimestamp:                                     Sun, 11 Nov 2018 07:46:29 +0000
Reference:                                             Deployment/linuxea-hpa
Metrics:                                               ( current / target )
  resource memory on pods:                             107750195200m / 50Mi
  resource cpu on pods  (as a percentage of request):  73% (221m) / 30%
Min replicas:                                          1
Max replicas:                                          5
Deployment pods:                                       5 current / 5 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     False   BackoffBoth       the time since the previous scale is still within both the downscale and upscale forbidden windows
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  True    TooManyReplicas   the desired replica count is more than the maximum replica count
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  2m    horizontal-pod-autoscaler  New size: 5; reason: memory resource above target

缩减的速度非常的慢，需要等待。

如果能够通过prometheus导出指标，就可以根据指标来做伸缩。每个pod上做的资源指标可以用否，取决于prometheus能够从 pod内的应用获取到什么指标的。在开发这些应用的时候就应该输出指标数据，当前能够承载的最大访问数，可以基于此类的方式，特定的指标做扩展。不过，这些需要进行自己去定义，并且支持。

自定义指标

type为pods，说明使用pods中的名称为http_requests的指标，达到300个就扩展

  metrics:
  - type: Pods
      name: 
        metricsName: http_requests
        targetAverageValue: 300m

其他参考