istio 故障注入/重试和容错/流量镜像(10)

2023年 7月 15日 29.7k 0

6.故障注入

istiO支持两种故障注入,分别是延迟故障和中断故障

  • 延迟故障:超时,重新发送请求
  • abort中断故障:重试

故障注入仍然在http层进行定义

中断故障

    fault:
      abort: # 中断故障
        percentage:
          value: 20     # 在多大的比例流量上注入
        httpStatus: 567 # 故障响应码

延迟故障

    fault:
      delay:
        percentage:
          value: 20    # 在百分之20的流量上注入
        fixedDelay: 6s # 注入三秒的延迟

yaml如下

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: dpment
  namespace: java-demo
spec:
  hosts:
  - "dpment.linuxea.com"                     # 对应于gateways/proxy-gateway
  - "dpment"
  gateways:
  - istio-system/dpment-gateway       # 相关定义仅应用于Ingress Gateway上  
  - mesh
  http:
  - name: version
    match:
    - uri:
        prefix: /version/
    rewrite:
      uri: /
    route:
    - destination:
        host: dpment
        subset: v10
    fault:
      abort:
        percentage:
          value: 20
        httpStatus: 567
  - name: default
    route:
    - destination:
        host: dpment
        subset: v11
    fault:
      delay:
        percentage:
          value: 20
        fixedDelay: 6s

此时,当我们用curl访问 dpment.linuxea.com的时候,有20的流量会被中断6秒

(base) [root@master1 7]# while true;do date;curl dpment.linuxea.com; date;sleep 0.$RANDOM;done
2022年 08月 07日 星期日 18:10:40 CST
linuxea-dpment-linuxea-b-55694cb7f5-lhkrb.com-127.0.0.1/8 130.130.1.122/24 version number 2.0
2022年 08月 07日 星期日 18:10:40 CST
2022年 08月 07日 星期日 18:10:41 CST
linuxea-dpment-linuxea-b-55694cb7f5-lhkrb.com-127.0.0.1/8 130.130.1.122/24 version number 2.0
2022年 08月 07日 星期日 18:10:41 CST
2022年 08月 07日 星期日 18:10:41 CST
linuxea-dpment-linuxea-b-55694cb7f5-lhkrb.com-127.0.0.1/8 130.130.1.122/24 version number 2.0
2022年 08月 07日 星期日 18:10:41 CST
2022年 08月 07日 星期日 18:10:41 CST
linuxea-dpment-linuxea-b-55694cb7f5-lhkrb.com-127.0.0.1/8 130.130.1.122/24 version number 2.0
2022年 08月 07日 星期日 18:10:47 CST
2022年 08月 07日 星期日 18:10:47 CST
linuxea-dpment-linuxea-b-55694cb7f5-lhkrb.com-127.0.0.1/8 130.130.1.122/24 version number 2.0
2022年 08月 07日 星期日 18:10:53 CST
2022年 08月 07日 星期日 18:10:54 CST
linuxea-dpment-linuxea-b-55694cb7f5-lhkrb.com-127.0.0.1/8 130.130.1.122/24 version number 2.0
2022年 08月 07日 星期日 18:10:54 CST
2022年 08月 07日 星期日 18:10:55 CST

image-20220807181349035.png

如果我们访问dpment.linuxea.com/version/的时候,有20%的流量返回的状态码是567

(base) [root@master1 7]# while true;do echo -e "===============";curl dpment.linuxea.com/version/   -I ; sleep 0.$RANDOM;done 
===============
HTTP/1.1 567 Unknown
content-length: 18
content-type: text/plain
date: Sun, 07 Aug 2022 10:16:40 GMT
server: istio-envoy

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:31 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 2

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:32 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1

===============
HTTP/1.1 567 Unknown
content-length: 18
content-type: text/plain
date: Sun, 07 Aug 2022 10:16:42 GMT
server: istio-envoy

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:33 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 3

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:33 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:33 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:33 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 3

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:34 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:34 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:35 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:35 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 2

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:36 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 2

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:36 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 2

===============
HTTP/1.1 567 Unknown
content-length: 18
content-type: text/plain
date: Sun, 07 Aug 2022 10:16:46 GMT
server: istio-envoy

===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:37 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1

如果使用curl命令直接访问会看到fault filter abort

(base) [root@master1 7]# while true;do echo -e "n";curl dpment.linuxea.com/version/    ; sleep 0.$RANDOM;done
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
fault filter abort
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
fault filter abort
fault filter abort
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
fault filter abort
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
fault filter abort
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0

回到kiali20220807-1.gif

6.1. 重试和容错

请求重试条件:

  • 5xx:上游主机返回5xx响应码,或者根本未响应(端口,重置,读取超时)
  • gateway-error: 网关错误,类似于5xx策略,但仅为502,503,504应用进行重试
  • connection-failure:在tcp级别与上游服务建立连接失败时进行重试
  • retriable-4xx:上游服务器返回可重复的4xx响应码时进行重试
  • refused-stream:上游服务器使用REFUSED-STREAM错误码重置时进行重试
  • retrable-status-codes:上游服务器的响应码与重试策略或者x-envoy-retriable-status-codes标头值中定义的响应码匹配时进行重试
  • reset:上游主机完全不响应(disconnect/reset/read超时),envoy将进行重试
  • retriable-headers:如果上游服务器响应报文匹配重试策略或x-envoy-retriable-header-names标头中包含的任何标头,则envoy将尝试重试
  • envoy-rateliited:标头中存在x-envoy-ratelimited时重试

重试条件2(同x-envoy-grpc-on标头):

  • cancelled: grpc应答标头中的状态码是"cancelled"时进行重试
  • deadline-exceeded: grpc应答标头中的状态码是"deadline-exceeded"时进行重试
  • internal: grpc应答标头中的状态码是“internal”时进行重试
  • resource-exhausted:grpc应答标头中的状态码是"resource-exhausted"时进行重试
  • unavailable:grpc应答标头中的状态码是“unavailable”时进行重试

默认情况下,envoy不会进行任何类型的重试操作,除非明确定义

我们假设现在有多个服务,A->B->C,A向后代理,或者访问其中的B出现了响应延迟,在A上配置容错机制,如下

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: dpment
  namespace: java-demo
spec:
  hosts:
  - "dpment.linuxea.com"                     # 对应于gateways/proxy-gateway
  - "dpment"
  gateways:
  - istio-system/dpment-gateway       # 相关定义仅应用于Ingress Gateway上  
  http:
  - name: default
    route:
    - destination:
        host: A
    timeout: 1s  # 如果上游超过1秒响应,就返回超时结果
    retries:     # 重试
      attempts: 5 # 重试次数
      perTryTimeout: 1s # 重试时间
      retryOn: 5xx,connect-failure,refused-stream # 对那些条件进行重试

如果上游服务超过1秒未响应就进行重试,对于5开头的响应码,tcp链接失败的,或者是GRPC的Refused-stream的建立链接也拒绝了,就重试五次,每次重试1秒。这个重试的 5次过程中,如果在1s内,有成功的则会成功 。

7.流量镜像

流量镜像,也叫影子流量(Traffic shadowing),是一种通过复制生产环境的流量到其他环境进行测试开发的工作模式。

在traffic-mirror中,我们可以直接使用mirror来指定给一个版本

  - name: default
    route:
    - destination:
        host: dpment
        subset: v11
    mirror:
      host: dpment
      subset: v12

于是,我们在此前的配置上修改

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: dpment
  namespace: java-demo
spec:
  hosts:
  - "dpment.linuxea.com"                     # 对应于gateways/proxy-gateway
  - "dpment"
  gateways:
  - istio-system/dpment-gateway       # 相关定义仅应用于Ingress Gateway上  
  - mesh
  http:
  - name: version
    match:
    - uri:
        prefix: /version/
    rewrite:
      uri: /
    route:
    - destination:
        host: dpment
        subset: v10
  - name: default
    route:
    - destination:
        host: dpment
        subset: v11
    mirror:
      host: dpment
      subset: v12

我们发起curl请求

 while ("true"){ curl http://dpment.linuxea.com/  ;sleep 1}

而后在v12中查看日志以获取是否流量被镜像进来

(base) [root@master1 10]# kubectl -n  java-demo exec -it dpment-linuxea-c-568b9fcb5c-ltdcg -- /bin/bash
bash-5.0# curl 127.0.0.1
linuxea-dpment-linuxea-c-568b9fcb5c-ltdcg.com-127.0.0.1/8 130.130.1.125/24 version number 3.0
bash-5.0# tail -f /data/logs/access.log 
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:27:59 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:00 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:01 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:02 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:03 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:04 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:05 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:06 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:07 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:08 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:11 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:12 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:13 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:14 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:15 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:16 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:17 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:18 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:19 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:20 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:21 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:23 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:24 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:25 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"

image-20220808122955725.png

相关文章

LeaferJS 1.0 重磅发布:强悍的前端 Canvas 渲染引擎
10分钟搞定支持通配符的永久有效免费HTTPS证书
300 多个 Microsoft Excel 快捷方式
一步步配置基于kubeadmin的kubevip高可用
istio全链路传递cookie和header灰度
REST Web 服务版本控制

发布评论