由于traefik 2.x和traefik 1.x的版本配置改动的比较多,这里先介绍1.x版本的配置。目前来说traefik 2.0是2019年12月份发布的,大部分企业还是使用1.x。我们跳槽去人家公司不能说光会2.x不会1.x版本,所以traefik会出2个版本的文档。目前我们使用的版本的1.7.20,为1.x版本的最后一个版本,更新时间为2019年12月10日
关于traefik 2.x的新特性在下一篇文章介绍
https://docs.traefik.io/migration/v1-to-v2/
Ingress 其实就是集群外部访问的一个入口,将外部的请求转发到不同的Server上,其实就相当于Nginx、Haproxy等负载均衡器。
Ingress实际上是通过服务发现的功能进行实现,通过Ingress controller来提供路由信息的刷新
Ingress controller可以理解为一个监视器,不断监听kube-apiserver,实时感知service、Pod的变化,Ingress controller再结合Ingress的配置,更新反向代理负载均衡器,达到服务发现的作用。
目前可以提供Ingress controller有很多,比如traefik、nginx-ingress、Kubernetes Ingress Cpmtrper for Kong、HAProxy Ingress controller等。
目前常见的负载均衡有Nginx-Ingress和traefik
这里使用traefik进行演示,主要是目前traefik已经成为主流。并且traefik有漂亮的dashboard界面,配置简单,已经已经深入和prometheus集成,nginx-ingress需要有3个组件镜像,traefik只有2个
Traefik
traefik是一款开源的反向代理和负载均衡工具,最大的优点是能够和常见的微服务系统直接整合,可以实现自动化动态配置。目前支持Docker、Swarm、Mesos、Mesos、Kubernetes、Consul、ETCD、Zookeeper等等后端模型
左边的是我们用户访问的域名,当请求到达traefik上,traefik会去监听API(kube-apiserver)监听访问的更新,这个简单的可以理解为Nginx的后端健康检查,实时监控Pod的状态。
核心概念
当启动Traefik时,需要定义entrypoints,然后通过entrypoints的路由来分析传入的请求,来查看他们是否是一组规则匹配,如果匹配,则路由可能将请求通过一系列的转换过来在发送到服务上去。
Traefik Install
使用traefik同样需要部署在Kubernetes集群当中
本次构建traefik的版本为v1.7.20
这里我们使用一台Node节点当做边缘节点
#环境说明 首先我们需要找到一个有外网网卡的服务器,我这里使用一台Node节点进行演示 服务器:192.168.0.13 (内网) 10.4.82.10 (外网) 网卡:eth0 内网 eth1 外网
这里首先我们需要先创建RBAC
apiVersion: v1 kind: ServiceAccount metadata: name: traefik-ingress-controller namespace: kube-system --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: traefik-ingress-controller rules: - apiGroups: - "" resources: - services - endpoints - secrets verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: traefik-ingress-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: traefik-ingress-controller subjects: - kind: ServiceAccount name: traefik-ingress-controller namespace: kube-system #这里直接设置名称创建就可以 [root@k8s-01 traefik]# kubectl apply -f traefik-rbac.yaml serviceaccount/traefik-ingress-controller created clusterrole.rbac.authorization.k8s.io/traefik-ingress-controller created clusterrolebinding.rbac.authorization.k8s.io/traefik-ingress-controller created [root@k8s-01 traefik]# kubectl get sa -n kube-system|grep traefik traefik-ingress-controller 1 35s
接下来我们使用修改Deployment文件
kind: Deployment apiVersion: extensions/v1beta1 metadata: name: traefik-ingress-controller namespace: kube-system labels: k8s-app: traefik-ingress-lb spec: replicas: 1 selector: matchLabels: k8s-app: traefik-ingress-lb template: metadata: labels: k8s-app: traefik-ingress-lb name: traefik-ingress-lb spec: serviceAccountName: traefik-ingress-controller terminationGracePeriodSeconds: 60 tolerations: - operator: "Exists" nodeSelector: kubernetes.io/hostname: k8s-04 containers: - image: traefik:v1.7.20 name: traefik-ingress-lb ports: - name: http containerPort: 80 - name: admin containerPort: 8080 args: - --api - --kubernetes - --logLevel=INFO --- kind: Service apiVersion: v1 metadata: name: traefik-ingress-service namespace: kube-system spec: selector: k8s-app: traefik-ingress-lb ports: - protocol: TCP port: 80 name: web - protocol: TCP port: 8080 name: admin type: NodePort [root@k8s-01 traefik]# kubectl apply -f traefik-deployment.yaml deployment.extensions/traefik-ingress-controller created service/traefik-ingress-service created
这里使用了nodeSelector标签将traefik固定到k8s-04节点上(k8s-04是双网卡,为了方便测试固定到一个节点上。否则就失去外网的作用)
还可以通过查看node节点标签进行获取
$ kubectl get node --show-labels k8s-01 Ready 5d3h v1.14.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-01,kubernetes.io/os=linux k8s-02 Ready 5d3h v1.14.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-02,kubernetes.io/os=linux k8s-03 Ready 5d3h v1.14.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-03,kubernetes.io/os=linux k8s-04 Ready 5d3h v1.14.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-04,kubernetes.io/os=linux
traefik提供了一个web ui,8080端口
[root@k8s-01 traefik]# kubectl get pod,svc -n kube-system -o wide|grep traefik pod/traefik-ingress-controller-7c59f4dd87-7qpxj 1/1 Running 0 2m54s 172.30.64.4 k8s-04 service/traefik-ingress-service NodePort 10.254.215.148 80:4577/TCP,8080:27237/TCP 2m54s k8s-app=traefik-ingress-lb
这里已经将8080映射到27237端口上,我们在集群任意一个机器访问这个端口就可以
FRONTENDS 相当于nginx server_name,表示添加的域名
BACKENDS 相当于我们proxy_pass,后端节点
配置Ingress对象
目前我们访问traefik使用的是NodePort方式,接下来我们创建一个ingress,使用域名的方式访问traefik
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: traefik-web namespace: kube-system annotations: kubernetes.io/ingress.class: traefik #表示在ingress中使用ingress controller spec: rules: #定义规则 - host: traefik.i4t.com #这里的host相当于nginx中的server_name http: paths: - backend: #后端,下面的serviceName代表我们之前创建的svc名称 serviceName: traefik-ingress-service servicePort: 8080 #service端口(这里也可以使用server port名称,之前traefik中已经定义了) [root@k8s-01 traefik]# kubectl apply -f traefik-ingress.yaml ingress.extensions/traefik-web created
创建完成后我们可以通过下面的命令进行查看
[root@k8s-01 traefik]# kubectl get ingress -n kube-system NAME HOSTS ADDRESS PORTS AGE traefik-web traefik.i4t.com 80 62s [root@k8s-01 traefik]# kubectl describe ingress -n kube-system traefik-web Name: traefik-web Namespace: kube-system Address: Default backend: default-http-backend:80 (<none>) Rules: Host Path Backends ---- ---- -------- traefik.i4t.com traefik-ingress-service:8080 (172.30.64.4:8080) Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"traefik"},"name":"traefik-web","namespace":"kube-system"},"spec":{"rules":[{"host":"traefik.i4t.com","http":{"paths":[{"backend":{"serviceName":"traefik-ingress-service","servicePort":8080}}]}}]}} kubernetes.io/ingress.class: traefik Events: #因为我们的ingress是在kube-system命名空间下面,查看的时候需要加上namespace参数
接下来我们做一下host解析
#mac 配置路径 ➜ ~ sudo -i Password: localhost:~ root# vim /etc/hosts localhost:~ root# tail -n 1 /etc/hosts 10.4.82.10 traefik.i4t.com #win 配置路径 C盘 -> Windows -> System32 -> drives -> etc -> hosts文件
由于我们8080(traefik web ui)配置的类型还是NodePort的形式,如果我们需要访问的话还是需要在域名的后面添加上端口
[root@k8s-01 traefik]# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.254.0.2 53/UDP,53/TCP,9153/TCP 5d3h kubernetes-dashboard NodePort 10.254.194.101 80:30000/TCP 5d3h traefik-ingress-service NodePort 10.254.215.148 80:4577/TCP,8080:27237/TCP 51m
我们可以点击HEALTH,查看我们的健康检查 (没有数据可以稍等一下)
因为目前我们访问还需要添加端口,非常的不方便。我们现在只需要修改我们traefik的deployment文件,在containers.ports添加一个hostPort端口
hostPort是直接将容器的端口与所调度的节点上的端口路由,这样用户就可以通过宿主机的IP加上来访问Pod
#执行下面的命令,或者修改原来的yaml文件,通过apply刷新也是可以的 $ kubectl edit deployment -n kube-system traefik-ingress-controller 添加 hostPort: 80
等Pod重启完毕,我们在访问进行查看
[root@k8s-01 traefik]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-6776c9d69-6chbp 1/1 Running 5 5d3h coredns-6776c9d69-f54n5 1/1 Running 4 5d3h kubernetes-dashboard-78bcf8c65c-vfbkq 1/1 Running 7 5d3h traefik-ingress-controller-688b8ffc8c-g4gvh 1/1 Running 0 39s
温馨提示: 我们修改的是Deployment,对svc的nodeport并没有什么影响
需要注意的一点,添加的nodePort域名解析必须在traefik Pod的服务器上
Traefik 配置HTTPS
证书我们很多时候使用的阿里云或者腾讯云的免费证书,当然如果觉得免费证书比较麻烦也可以使用openssl命令生成证书 (在traefik 2.x版本中已经支持使用Let's Encrypt自动生成证书),这里还是使用openssl生成证书进行演示
当然生产环境肯定是用CA机构颁发的证书,测试环境我就使用openssl生成了
#创建一个临时证书目录 mkdir /root/traefik-ssl/ && cd /root/traefik-ssl #创建证书 openssl req -newkey rsa:2048 -nodes -keyout tls.key -x509 -days 365 -out tls.crt #证书时间365 #这里信息都无所谓,直接一路回车就可以 #当期目录下会产生一个crt文件和一个key文件 [root@k8s-01 traefik-ssl]# ls tls.crt tls.key
这里我们需要使用secret对象,将证书储存到secret中
kubectl create secret generic traefik-cert --from-file=tls.crt --from-file=tls.key -n kube-system #如果有现成的证书文件,修改个名称就可以使用了,步骤不变
参数解释
创建完成后可以通过下面的命令进行检查
[root@k8s-01 traefik-ssl]# kubectl get secret -n kube-system|grep traefik traefik-cert Opaque 2 108s traefik-ingress-controller-token-mnnq6 kubernetes.io/service-account-token 3 96m [root@k8s-01 traefik-ssl]# kubectl describe secret -n kube-system traefik-cert Name: traefik-cert Namespace: kube-system Labels: Annotations: Type: Opaque Data ==== tls.crt: 1220 bytes tls.key: 1704 bytes
这里需要给traefik创建一个配置文件 (这里配置文件的后缀为toml)
$ vim /root/traefik-ssl/traefik.toml defaultEntryPoints = ['http', 'https'] [entryPoints] #指定一个入口 [entryPoints.http] #配置一个http的入口 address = ":80" #http的端口 (根据svc自行修改) [entryPoints.http.redirect] #强制将http协议跳转到https,相当于nginx 301跳转。不添加也可以 entryPoint = 'https' #因为强制跳转到https上,接下来需要配置https [entryPoints.https] address = ":443" [entryPoints.https.tls] #https需要指定2个证书 [[entryPoints.https.tls.certificates]] CertFile = "/ssl/tls.crt" #证书文件目录 KeyFile = "/ssl/tls.key" #这里需要将文件保存为traefik.toml
现在配置文件在本地,我们需要创建一个configmap,使用configmap的方式挂载到traefik中
kubectl create configmap traefik-conf --from-file=/root/traefik-ssl/traefik.toml -n kube-system
创建完成可以检查一下
关于configmap可以参考https://k.i4t.com/kubernetes_configmap.html
修改traefik的Deployment文件,将configmap和secret挂载到traefik容器中,并且修改traefik启动参数、添加443端口
#注释位置代表添加位置,如果名称没有发生变化。可以直接复制我的配置文件更新 kind: Deployment apiVersion: extensions/v1beta1 metadata: name: traefik-ingress-controller namespace: kube-system labels: k8s-app: traefik-ingress-lb spec: replicas: 1 selector: matchLabels: k8s-app: traefik-ingress-lb template: metadata: labels: k8s-app: traefik-ingress-lb name: traefik-ingress-lb spec: serviceAccountName: traefik-ingress-controller terminationGracePeriodSeconds: 60 volumes: #这里的Mount相当于声明 - name: ssl secret: #引用之前创建的secret secretName: traefik-cert - name: config configMap: #这里引用configmap资源对象 name: traefik-conf tolerations: - operator: "Exists" nodeSelector: kubernetes.io/hostname: k8s-04 containers: - image: traefik:v1.7.20 name: traefik-ingress-lb volumeMounts: #将声明的文件挂载到Pod路径 - mountPath: "/ssl" name: ssl #将secret资源对象挂载到/ssl目录中 - mountPath: "/config" #将configmap声明的对象挂载到/config下 name: config ports: - name: http containerPort: 80 hostPort: 80 - name: https #为pod添加一个443端口 containerPort: 443 hostPort: 443 - name: admin containerPort: 8080 args: - --configfile=/config/traefik.toml #设置启动参数,指定配置文件 - --api - --kubernetes - --logLevel=INFO --- kind: Service apiVersion: v1 metadata: name: traefik-ingress-service namespace: kube-system spec: selector: k8s-app: traefik-ingress-lb ports: - protocol: TCP port: 80 name: web - protocol: TCP port: 8080 name: admin type: NodePort #创建更新 [root@k8s-01 traefik]# kubectl apply -f traefik-deployment.yaml deployment.extensions/traefik-ingress-controller configured service/traefik-ingress-service unchanged
检查Pod运行状态
kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-6776c9d69-6chbp 1/1 Running 5 5d4h coredns-6776c9d69-f54n5 1/1 Running 4 5d4h kubernetes-dashboard-78bcf8c65c-vfbkq 1/1 Running 7 5d4h traefik-ingress-controller-7c46b4b8f5-zbfzf 1/1 Running 0 31s
这时候我们在强制刷新,打开配置的域名,就可以看到traefik已经强制将80跳转到443。因为证书没有CA机构签署,所以会提示我们不安全
并且查看证书是无效的
配置traefik path
上面简单说了一下TLS使用方法,这里使用tomcat部署访问,使用path功能,相当于nginx location功能
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: tomcat1 spec: replicas: 1 template: metadata: labels: app: tomcat1 spec: containers: - name: tomcat1 image: abcdocker9/tomcat:v1.1 ports: - containerPort: 8080 protocol: TCP --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: tomcat2 spec: replicas: 1 template: metadata: labels: app: tomcat2 spec: containers: - name: tomcat2 image: abcdocker9/tomcat:v1.2 ports: - containerPort: 8080 protocol: TCP --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: tomcat3 spec: replicas: 1 template: metadata: labels: app: tomcat3 spec: containers: - name: tomcat3 image: abcdocker9/tomcat:v1.3 ports: - containerPort: 8080 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app: tomcat1 name: tomcat1 spec: type: ClusterIP ports: - name: http port: 8080 selector: app: tomcat1 --- apiVersion: v1 kind: Service metadata: labels: app: tomcat2 name: tomcat2 spec: type: ClusterIP ports: - name: http port: 8080 selector: app: tomcat2 --- apiVersion: v1 kind: Service metadata: labels: app: tomcat3 name: tomcat3 spec: type: ClusterIP ports: - name: http port: 8080 selector: app: tomcat3 #这里主要创建了3个deployment,镜像为tomcat,并且设置了svc [root@k8s-01 traefik]# kubectl apply -f test.yaml deployment.extensions/tomcat1 created deployment.extensions/tomcat2 created deployment.extensions/tomcat3 created service/tomcat1 created service/tomcat2 created service/tomcat3 created #镜像上传到docker的镜像仓库,直接下载就可以,网速慢的等一会就可以了
接下来我们查看一下Pod状态
[root@k8s-01 traefik]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-deploy-56db997f77-74qfj 1/1 Running 2 2d12h 172.30.40.4 k8s-01 nginx-deploy-56db997f77-khv26 1/1 Running 2 2d12h 172.30.128.3 k8s-02 nginx-ds-6sbj8 1/1 Running 4 5d16h 172.30.128.2 k8s-02 nginx-ds-q7h76 1/1 Running 5 5d16h 172.30.24.4 k8s-03 nginx-ds-vnv6s 1/1 Running 6 5d16h 172.30.64.2 k8s-04 nginx-ds-vtx6m 1/1 Running 4 5d16h 172.30.40.2 k8s-01 tomcat1-9c8ffd457-d2q9t 1/1 Running 0 8m1s 172.30.40.6 k8s-01 tomcat2-79df4f4b58-mkhbl 1/1 Running 0 8m1s 172.30.24.3 k8s-03 tomcat3-79f5b65b9f-vb2bn 1/1 Running 0 8m1s 172.30.24.5 k8s-03 #查看一下svc [root@k8s-01 traefik]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.254.0.1 443/TCP 5d17h nginx-ds NodePort 10.254.86.215 80:6339/TCP 5d16h nginx-service ClusterIP 10.254.159.126 18080/TCP 2d12h tomcat1 ClusterIP 10.254.179.147 8080/TCP 8m35s tomcat2 ClusterIP 10.254.183.15 8080/TCP 8m35s tomcat3 ClusterIP 10.254.229.191 8080/TCP 8m35s [root@k8s-01 traefik]# curl 10.254.179.147:8080 abcdocker_01 [root@k8s-01 traefik]# curl 10.254.183.15:8080 abcdocker_02 [root@k8s-01 traefik]# curl 10.254.229.191:8080 abcdocker_03 #测试都是没有问题的,就可以继续下一步
创建ingress对象
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: tomcat-web #ingress名称 annotations: #指定ingress类型为traefik kubernetes.io/ingress.class: traefik traefik.frontend.rule.type: PathPrefixStrip ##跳转后端时忽略 path (如果不添加这个参数,我们请求到v1的时候,在v1访问依旧会带有v1) spec: rules: #配置规则 - host: tomcat.i4t.com http: paths: #通过path来区分不同的环境 - path: /v1 backend: serviceName: tomcat1 #指定svc 名称 servicePort: 8080 #svc 端口 - path: /v2 backend: serviceName: tomcat2 servicePort: 8080 - path: / #这里代表默认/,不加任何参数访问的的svc backend: serviceName: tomcat3 servicePort: 8080 #这里需要将我们的path /放到最后,这里的path也是优先级的,如果将/放到最上面,很有可能无法匹配到下面的v1和v2 [root@k8s-01 traefik]# kubectl apply -f test-ingress.yaml ingress.extensions/tomcat-web created
这里创建的ingress没有指定命名空间,默认在default下,直接就可以查看
[root@k8s-01 traefik]# kubectl get ingress NAME HOSTS ADDRESS PORTS AGE tomcat-web tomcat.i4t.com 80 24s [root@k8s-01 traefik]# kubectl describe ingress tomcat-web Name: tomcat-web Namespace: default Address: Default backend: default-http-backend:80 (<none>) Rules: Host Path Backends ---- ---- -------- tomcat.i4t.com /v1 tomcat1:8080 (172.30.40.6:8080) /v2 tomcat2:8080 (172.30.24.3:8080) / tomcat3:8080 (172.30.128.4:8080) Annotations: kubernetes.io/ingress.class: traefik kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"traefik"},"name":"tomcat-web","namespace":"default"},"spec":{"rules":[{"host":"tomcat.i4t.com","http":{"paths":[{"backend":{"serviceName":"tomcat1","servicePort":8080},"path":"/v1"},{"backend":{"serviceName":"tomcat2","servicePort":8080},"path":"/v2"},{"backend":{"serviceName":"tomcat3","servicePort":8080},"path":"/"}]}}]}} Events: #这里可以看到我们的域名已经有对应的Path,以及svc名称和端口,括号里面的是pod ip
现在我们添加域名解析就可以访问了,这里还是修改host为主
vim /etc/hosts 10.4.82.10 tomcat.i4t.com #这里我们还是解析到公网地址(traefik在node4节点,node4内外网前面环境说明已经写了)
现在我们查看traefik,可以看到数据已经刷新
接下来可以访问测试
接下来配置一下https
我们traefik已经强制跳转了https,但是有可能我们的证书不是traefik域名的。现在我们就当证书已经下载下来,从新创建一个secret
kubectl create secret generic traefik-ssl --from-file=tls.crt --from-file=tls.key #这次取名叫traefik-ssl 命名空间和tomcat-web在同一个下面
修改ingress文件
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: tomcat-web annotations: kubernetes.io/ingress.class: traefik traefik.frontend.rule.type: PathPrefixStrip spec: tls: #这里添加我们的secret证书,这里我们还是用之前的证书 - secretName: traefik-cert #这里是secret的名称,也就是我们挂载证书的地方,我这里使用之前创建的,如果新的只需要从新挂载一下设置一下名称就可以 rules: - host: tomcat.i4t.com http: paths: - path: /v1 backend: serviceName: tomcat1 #指定svc 名称 servicePort: 8080 #svc 端口 - path: /v2 backend: serviceName: tomcat2 servicePort: 8080 - path: / backend: serviceName: tomcat3 servicePort: 8080
接下来我们刷新配置文件
[root@k8s-01 traefik]# kubectl apply -f test-ingress.yaml ingress.extensions/tomcat-web configured
访问查看
相关文章:
- Kubernetes 1.14 二进制集群安装
- Kuerbernetes 1.11 集群二进制安装
- Kubenetes 1.13.5 集群二进制安装
- CentOS 7 ETCD集群配置大全