6.故障注入
istiO支持两种故障注入,分别是延迟故障和中断故障
- 延迟故障:超时,重新发送请求
- abort中断故障:重试
故障注入仍然在http层进行定义
中断故障
fault:
abort: # 中断故障
percentage:
value: 20 # 在多大的比例流量上注入
httpStatus: 567 # 故障响应码
延迟故障
fault:
delay:
percentage:
value: 20 # 在百分之20的流量上注入
fixedDelay: 6s # 注入三秒的延迟
yaml如下
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: dpment
namespace: java-demo
spec:
hosts:
- "dpment.linuxea.com" # 对应于gateways/proxy-gateway
- "dpment"
gateways:
- istio-system/dpment-gateway # 相关定义仅应用于Ingress Gateway上
- mesh
http:
- name: version
match:
- uri:
prefix: /version/
rewrite:
uri: /
route:
- destination:
host: dpment
subset: v10
fault:
abort:
percentage:
value: 20
httpStatus: 567
- name: default
route:
- destination:
host: dpment
subset: v11
fault:
delay:
percentage:
value: 20
fixedDelay: 6s
此时,当我们用curl访问 dpment.linuxea.com的时候,有20的流量会被中断6秒
(base) [root@master1 7]# while true;do date;curl dpment.linuxea.com; date;sleep 0.$RANDOM;done
2022年 08月 07日 星期日 18:10:40 CST
linuxea-dpment-linuxea-b-55694cb7f5-lhkrb.com-127.0.0.1/8 130.130.1.122/24 version number 2.0
2022年 08月 07日 星期日 18:10:40 CST
2022年 08月 07日 星期日 18:10:41 CST
linuxea-dpment-linuxea-b-55694cb7f5-lhkrb.com-127.0.0.1/8 130.130.1.122/24 version number 2.0
2022年 08月 07日 星期日 18:10:41 CST
2022年 08月 07日 星期日 18:10:41 CST
linuxea-dpment-linuxea-b-55694cb7f5-lhkrb.com-127.0.0.1/8 130.130.1.122/24 version number 2.0
2022年 08月 07日 星期日 18:10:41 CST
2022年 08月 07日 星期日 18:10:41 CST
linuxea-dpment-linuxea-b-55694cb7f5-lhkrb.com-127.0.0.1/8 130.130.1.122/24 version number 2.0
2022年 08月 07日 星期日 18:10:47 CST
2022年 08月 07日 星期日 18:10:47 CST
linuxea-dpment-linuxea-b-55694cb7f5-lhkrb.com-127.0.0.1/8 130.130.1.122/24 version number 2.0
2022年 08月 07日 星期日 18:10:53 CST
2022年 08月 07日 星期日 18:10:54 CST
linuxea-dpment-linuxea-b-55694cb7f5-lhkrb.com-127.0.0.1/8 130.130.1.122/24 version number 2.0
2022年 08月 07日 星期日 18:10:54 CST
2022年 08月 07日 星期日 18:10:55 CST
如果我们访问dpment.linuxea.com/version/的时候,有20%的流量返回的状态码是567
(base) [root@master1 7]# while true;do echo -e "===============";curl dpment.linuxea.com/version/ -I ; sleep 0.$RANDOM;done
===============
HTTP/1.1 567 Unknown
content-length: 18
content-type: text/plain
date: Sun, 07 Aug 2022 10:16:40 GMT
server: istio-envoy
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:31 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 2
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:32 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1
===============
HTTP/1.1 567 Unknown
content-length: 18
content-type: text/plain
date: Sun, 07 Aug 2022 10:16:42 GMT
server: istio-envoy
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:33 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 3
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:33 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:33 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:33 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 3
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:34 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:34 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:35 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:35 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 2
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:36 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 2
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:36 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 2
===============
HTTP/1.1 567 Unknown
content-length: 18
content-type: text/plain
date: Sun, 07 Aug 2022 10:16:46 GMT
server: istio-envoy
===============
HTTP/1.1 200 OK
server: istio-envoy
date: Sun, 07 Aug 2022 10:17:37 GMT
content-type: text/html
content-length: 93
last-modified: Wed, 03 Aug 2022 07:59:37 GMT
etag: "62ea2ae9-5d"
accept-ranges: bytes
x-envoy-upstream-service-time: 1
如果使用curl命令直接访问会看到fault filter abort
(base) [root@master1 7]# while true;do echo -e "n";curl dpment.linuxea.com/version/ ; sleep 0.$RANDOM;done
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
fault filter abort
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
fault filter abort
fault filter abort
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
fault filter abort
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
fault filter abort
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
linuxea-dpment-linuxea-a-777847fd74-fsnsv.com-127.0.0.1/8 130.130.0.19/24 version number 1.0
回到kiali
6.1. 重试和容错
请求重试条件:
- 5xx:上游主机返回5xx响应码,或者根本未响应(端口,重置,读取超时)
- gateway-error: 网关错误,类似于5xx策略,但仅为502,503,504应用进行重试
- connection-failure:在tcp级别与上游服务建立连接失败时进行重试
- retriable-4xx:上游服务器返回可重复的4xx响应码时进行重试
- refused-stream:上游服务器使用REFUSED-STREAM错误码重置时进行重试
- retrable-status-codes:上游服务器的响应码与重试策略或者x-envoy-retriable-status-codes标头值中定义的响应码匹配时进行重试
- reset:上游主机完全不响应(disconnect/reset/read超时),envoy将进行重试
- retriable-headers:如果上游服务器响应报文匹配重试策略或x-envoy-retriable-header-names标头中包含的任何标头,则envoy将尝试重试
- envoy-rateliited:标头中存在x-envoy-ratelimited时重试
重试条件2(同x-envoy-grpc-on标头):
- cancelled: grpc应答标头中的状态码是"cancelled"时进行重试
- deadline-exceeded: grpc应答标头中的状态码是"deadline-exceeded"时进行重试
- internal: grpc应答标头中的状态码是“internal”时进行重试
- resource-exhausted:grpc应答标头中的状态码是"resource-exhausted"时进行重试
- unavailable:grpc应答标头中的状态码是“unavailable”时进行重试
默认情况下,envoy不会进行任何类型的重试操作,除非明确定义
我们假设现在有多个服务,A->B->C,A向后代理,或者访问其中的B出现了响应延迟,在A上配置容错机制,如下
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: dpment
namespace: java-demo
spec:
hosts:
- "dpment.linuxea.com" # 对应于gateways/proxy-gateway
- "dpment"
gateways:
- istio-system/dpment-gateway # 相关定义仅应用于Ingress Gateway上
http:
- name: default
route:
- destination:
host: A
timeout: 1s # 如果上游超过1秒响应,就返回超时结果
retries: # 重试
attempts: 5 # 重试次数
perTryTimeout: 1s # 重试时间
retryOn: 5xx,connect-failure,refused-stream # 对那些条件进行重试
如果上游服务超过1秒未响应就进行重试,对于5开头的响应码,tcp链接失败的,或者是GRPC的Refused-stream的建立链接也拒绝了,就重试五次,每次重试1秒。这个重试的 5次过程中,如果在1s内,有成功的则会成功 。
7.流量镜像
流量镜像,也叫影子流量(Traffic shadowing),是一种通过复制生产环境的流量到其他环境进行测试开发的工作模式。
在traffic-mirror中,我们可以直接使用mirror来指定给一个版本
- name: default
route:
- destination:
host: dpment
subset: v11
mirror:
host: dpment
subset: v12
于是,我们在此前的配置上修改
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: dpment
namespace: java-demo
spec:
hosts:
- "dpment.linuxea.com" # 对应于gateways/proxy-gateway
- "dpment"
gateways:
- istio-system/dpment-gateway # 相关定义仅应用于Ingress Gateway上
- mesh
http:
- name: version
match:
- uri:
prefix: /version/
rewrite:
uri: /
route:
- destination:
host: dpment
subset: v10
- name: default
route:
- destination:
host: dpment
subset: v11
mirror:
host: dpment
subset: v12
我们发起curl请求
while ("true"){ curl http://dpment.linuxea.com/ ;sleep 1}
而后在v12中查看日志以获取是否流量被镜像进来
(base) [root@master1 10]# kubectl -n java-demo exec -it dpment-linuxea-c-568b9fcb5c-ltdcg -- /bin/bash
bash-5.0# curl 127.0.0.1
linuxea-dpment-linuxea-c-568b9fcb5c-ltdcg.com-127.0.0.1/8 130.130.1.125/24 version number 3.0
bash-5.0# tail -f /data/logs/access.log
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:27:59 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:00 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:01 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:02 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:03 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:04 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:05 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:06 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:07 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:08 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:11 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:12 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:13 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:14 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:15 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:16 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:17 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:18 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:19 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:20 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:21 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:23 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:24 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"
130.130.0.0,130.130.1.96, 127.0.0.6 - [08/Aug/2022:04:28:25 +0000] "GET / HTTP/1.1" dpment.linuxea.com-shadow94 "-" "curl/7.83.1" - -0.000 [200] [-] [-] "-"