k8s HELM 安装Kafka Zookeeper集群

云计算 2023-07-27 张二河 手机阅读

很早之前文章介绍了通过Kafka 二进制安装集群，目前很多环境都是Kubernetes，追求快速部署、快速创建项目。 下面我们通过helm快速构建一套Kafka集群并配置持久化

关于k8s sc持久化和Kafka二进制安装，此处就不在介绍了，可以参考下面的文章

Kafka容器化会受底层物理机的配置影响，大并发常景还是要慎重考虑

Helm 安装

相关服务版本

  • Kubernetes 1.24.0
  • Containerd v1.6.4
  1. <span># 下载</span>
    2. 

  2. <span>wget https</span>
<span>:</span>
<span>//get.helm.sh/helm-v3.6.1-linux-amd64.tar.gz</span>
    3. 

    4. 

  4. <span># 解压</span>
    5. 

  5. <span>tar zxvf helm</span>
<span>-</span>
<span>v3</span>
<span>.</span>
<span>6.1</span>
<span>-</span>
<span>linux</span>
<span>-</span>
<span>amd64</span>
<span>.</span>
<span>tar</span>
<span>.</span>
<span>gz</span>
    6. 

    7. 

  7. <span># 安装</span>
    8. 

  8. <span>mv linux</span>
<span>-</span>
<span>amd64</span>
<span>/</span>
<span>helm </span>
<span>/</span>
<span>usr</span>
<span>/</span>
<span>local</span>
<span>/</span>
<span>bin</span>
<span>/</span>
    9. 

    10. 

  10. <span># 验证</span>
    11. 

  11. <span>helm version</span>

Helm 部署Zookeeper集群

  1. <span># 添加bitnami仓库</span>
    2. 

  2. <span>helm repo add bitnami https</span>
<span>:</span>
<span>//charts.bitnami.com/bitnami</span>
    3. 

    4. 

  4. <span># 查询chart</span>
    5. 

  5. <span>helm search repo bitnami</span>
    6. 

    7. 

  7. <span># 拉取zookeeper</span>
    8. 

  8. <span>helm pull bitnami</span>
<span>/</span>
<span>zookeeper</span>
    9. 

    10. 

  10. <span># 解压</span>
    11. 

  11. <span>tar zxvf zookeeper</span>
<span>-</span>
<span>11.4</span>
<span>.</span>
<span>2.tgz</span>
    12. 

    13. 

  13. <span>#进入Zookeeper</span>
    14. 

  14. <span>cd zookeeper</span>

接下来对Zookeeper进行时区、持久化存储、副本数等配置

  1. <span>extraEnvVars</span>
<span>:</span>
<span> </span>
    2. 

  2. <span>  </span>
<span>-</span>
<span> name</span>
<span>:</span>
<span> TZ</span>
    3. 

  3. <span>    value</span>
<span>:</span>
<span> </span>
<span>"Asia/Shanghai"</span>
    4. 

    5. 

  5. <span># 允许任意用户连接（默认开启）</span>
    6. 

  6. <span>allowAnonymousLogin</span>
<span>:</span>
<span> </span>
<span>true</span>
    7. 

  7. <span>---</span>
    8. 

    9. 

  9. <span># 关闭认证（默认关闭）</span>
    10. 

  10. <span>auth</span>
<span>:</span>
    11. 

  11. <span>  enable</span>
<span>:</span>
<span> </span>
<span>false</span>
<span> </span>
    12. 

  12. <span>---</span>
    13. 

    14. 

  14. <span># 修改副本数</span>
    15. 

  15. <span>replicaCount</span>
<span>:</span>
<span> </span>
<span>3</span>
<span> </span>
    16. 

  16. <span>---</span>
    17. 

    18. 

  18. <span># 4. 配置持久化，按需使用</span>
    19. 

  19. <span>persistence</span>
<span>:</span>
    20. 

  20. <span>  enabled</span>
<span>:</span>
<span> </span>
<span>true</span>
    21. 

  21. <span>  storageClass</span>
<span>:</span>
<span> </span>
<span>"rook-ceph-block"</span>
<span>  </span>
<span># storageClass 如果有默认存储可以不写</span>
    22. 

  22. <span>  accessModes</span>
<span>:</span>
    23. 

  23. <span>    </span>
<span>-</span>
<span> </span>
<span>ReadWriteOnce</span>

创建Kafka namespace

  1. <span>[</span>
<span>root@k8s</span>
<span>-</span>
<span>02</span>
<span> zookeeper</span>
<span>]#</span>
<span> kubectl create ns kafka</span>

helm创建Zookeeper集群

  1. <span>[</span>
<span>root@k8s</span>
<span>-</span>
<span>02</span>
<span> zookeeper</span>
<span>]#</span>
<span> helm install zookeeper </span>
<span>-</span>
<span>n kafka </span>
<span>.</span>
    2. 

  2. <span>#此处下面环境版本信息</span>
    3. 

  3. <span>NAME</span>
<span>:</span>
<span> zookeeper</span>
    4. 

  4. <span>LAST DEPLOYED</span>
<span>:</span>
<span> </span>
<span>Tue</span>
<span> </span>
<span>May</span>
<span> </span>
<span>23</span>
<span> </span>
<span>13</span>
<span>:</span>
<span>40</span>
<span>:</span>
<span>12</span>
<span> </span>
<span>2023</span>
    5. 

  5. <span>NAMESPACE</span>
<span>:</span>
<span> kafka</span>
    6. 

  6. <span>STATUS</span>
<span>:</span>
<span> deployed</span>
    7. 

  7. <span>REVISION</span>
<span>:</span>
<span> </span>
<span>1</span>
    8. 

  8. <span>TEST SUITE</span>
<span>:</span>
<span> </span>
<span>None</span>
    9. 

  9. <span>NOTES</span>
<span>:</span>
    10. 

  10. <span>CHART NAME</span>
<span>:</span>
<span> zookeeper</span>
    11. 

  11. <span>CHART VERSION</span>
<span>:</span>
<span> </span>
<span>11.4</span>
<span>.</span>
<span>2</span>
    12. 

  12. <span>APP VERSION</span>
<span>:</span>
<span> </span>
<span>3.8</span>
<span>.</span>
<span>1</span>
    13. 

    14. 

  14. <span>#下面为相关注释，后面可以通过下面的命令查看Zookeeper集群状态</span>
    15. 

  15. <span>**</span>
<span> </span>
<span>Please</span>
<span> be patient </span>
<span>while</span>
<span> the chart </span>
<span>is</span>
<span> being deployed </span>
<span>**</span>
    16. 

    17. 

  17. <span>ZooKeeper</span>
<span> can be accessed via port </span>
<span>2181</span>
<span> on the following DNS name </span>
<span>from</span>
<span> within your cluster</span>
<span>:</span>
    18. 

    19. 

  19. <span>    zookeeper</span>
<span>.</span>
<span>kafka</span>
<span>.</span>
<span>svc</span>
<span>.</span>
<span>cluster</span>
<span>.</span>
<span>local</span>
    20. 

    21. 

  21. <span>To</span>
<span> connect to your </span>
<span>ZooKeeper</span>
<span> server run the following commands</span>
<span>:</span>
    22. 

    23. 

  23. <span>    </span>
<span>export</span>
<span> POD_NAME</span>
<span>=</span>
<span>$</span>
<span>(</span>
<span>kubectl </span>
<span>get</span>
<span> pods </span>
<span>--</span>
<span>namespace</span>
<span> kafka </span>
<span>-</span>
<span>l </span>
<span>"app.kubernetes.io/name=zookeeper,app.kubernetes.io/instance=zookeeper,app.kubernetes.io/component=zookeeper"</span>
<span> </span>
<span>-</span>
<span>o jsonpath</span>
<span>=</span>
<span>"{.items[0].metadata.name}"</span>
<span>)</span>
    24. 

  24. <span>    kubectl </span>
<span>exec</span>
<span> </span>
<span>-</span>
<span>it $POD_NAME </span>
<span>--</span>
<span> zkCli</span>
<span>.</span>
<span>sh</span>
    25. 

    26. 

  26. <span>To</span>
<span> connect to your </span>
<span>ZooKeeper</span>
<span> server </span>
<span>from</span>
<span> outside the cluster execute the following commands</span>
<span>:</span>
    27. 

    28. 

  28. <span>    kubectl port</span>
<span>-</span>
<span>forward </span>
<span>--</span>
<span>namespace</span>
<span> kafka svc</span>
<span>/</span>
<span>zookeeper </span>
<span>2181</span>
<span>:</span>
<span>2181</span>
<span> </span>
<span>&</span>
    29. 

  29. <span>    zkCli</span>
<span>.</span>
<span>sh </span>
<span>127.0</span>
<span>.</span>
<span>0.1</span>
<span>:</span>
<span>2181</span>
    30. 

  30. <span>[</span>
<span>root@k8s</span>
<span>-</span>
<span>02</span>
<span> zookeeper</span>
<span>]#</span>

检查pod状态

  1. <span>[</span>
<span>root@k8s</span>
<span>-</span>
<span>02</span>
<span> zookeeper</span>
<span>]#</span>
<span> kubectl </span>
<span>get</span>
<span> all </span>
<span>-</span>
<span>n kafka</span>
    2. 

  2. <span>NAME              READY   STATUS    RESTARTS   AGE</span>
    3. 

  3. <span>pod</span>
<span>/</span>
<span>zookeeper</span>
<span>-</span>
<span>0</span>
<span>   </span>
<span>1</span>
<span>/</span>
<span>1</span>
<span>     </span>
<span>Running</span>
<span>   </span>
<span>0</span>
<span>          </span>
<span>52s</span>
    4. 

  4. <span>pod</span>
<span>/</span>
<span>zookeeper</span>
<span>-</span>
<span>1</span>
<span>   </span>
<span>1</span>
<span>/</span>
<span>1</span>
<span>     </span>
<span>Running</span>
<span>   </span>
<span>0</span>
<span>          </span>
<span>51s</span>
    5. 

  5. <span>pod</span>
<span>/</span>
<span>zookeeper</span>
<span>-</span>
<span>2</span>
<span>   </span>
<span>1</span>
<span>/</span>
<span>1</span>
<span>     </span>
<span>Running</span>
<span>   </span>
<span>0</span>
<span>          </span>
<span>49s</span>
    6. 

    7. 

  7. <span>NAME                         TYPE        CLUSTER</span>
<span>-</span>
<span>IP       EXTERNAL</span>
<span>-</span>
<span>IP   PORT</span>
<span>(</span>
<span>S</span>
<span>)</span>
<span>                      AGE</span>
    8. 

  8. <span>service</span>
<span>/</span>
<span>zookeeper            </span>
<span>ClusterIP</span>
<span>   </span>
<span>10.110</span>
<span>.</span>
<span>142.203</span>
<span>   </span>
<span><none></span>
<span>        </span>
<span>2181</span>
<span>/</span>
<span>TCP</span>
<span>,</span>
<span>2888</span>
<span>/</span>
<span>TCP</span>
<span>,</span>
<span>3888</span>
<span>/</span>
<span>TCP   </span>
<span>52s</span>
    9. 

  9. <span>service</span>
<span>/</span>
<span>zookeeper</span>
<span>-</span>
<span>headless   </span>
<span>ClusterIP</span>
<span>   </span>
<span>None</span>
<span>             </span>
<span><none></span>
<span>        </span>
<span>2181</span>
<span>/</span>
<span>TCP</span>
<span>,</span>
<span>2888</span>
<span>/</span>
<span>TCP</span>
<span>,</span>
<span>3888</span>
<span>/</span>
<span>TCP   </span>
<span>52s</span>
    10. 

    11. 

  11. <span>NAME                         READY   AGE</span>
    12. 

  12. <span>statefulset</span>
<span>.</span>
<span>apps</span>
<span>/</span>
<span>zookeeper   </span>
<span>3</span>
<span>/</span>
<span>3</span>
<span>     </span>
<span>52s</span>

检查pvc

  1. <span>[</span>
<span>root@k8s</span>
<span>-</span>
<span>02</span>
<span> zookeeper</span>
<span>]#</span>
<span> kubectl </span>
<span>get</span>
<span> pvc  </span>
<span>|</span>
<span>grep zook</span>
    2. 

  2. <span>data</span>
<span>-</span>
<span>zookeeper</span>
<span>-</span>
<span>0</span>
<span>   </span>
<span>Bound</span>
<span>    pvc</span>
<span>-</span>
<span>997a81c1</span>
<span>-</span>
<span>6986</span>
<span>-</span>
<span>4620</span>
<span>-</span>
<span>88f4</span>
<span>-</span>
<span>2270247354f5</span>
<span>   </span>
<span>8Gi</span>
<span>        RWO            nfs</span>
<span>-</span>
<span>storage    </span>
<span>7d23h</span>
    3. 

  3. <span>data</span>
<span>-</span>
<span>zookeeper</span>
<span>-</span>
<span>1</span>
<span>   </span>
<span>Bound</span>
<span>    pvc</span>
<span>-</span>
<span>a6012ebb</span>
<span>-</span>
<span>1f70</span>
<span>-</span>
<span>43d1</span>
<span>-</span>
<span>ac1b</span>
<span>-</span>
<span>8deaec660efe</span>
<span>   </span>
<span>8Gi</span>
<span>        RWO            nfs</span>
<span>-</span>
<span>storage    </span>
<span>7d23h</span>
    4. 

  4. <span>data</span>
<span>-</span>
<span>zookeeper</span>
<span>-</span>
<span>2</span>
<span>   </span>
<span>Bound</span>
<span>    pvc</span>
<span>-</span>
<span>f6300de4</span>
<span>-</span>
<span>8cd9</span>
<span>-</span>
<span>4807</span>
<span>-</span>
<span>a5fd</span>
<span>-</span>
<span>2655deb05139</span>
<span>   </span>
<span>8Gi</span>
<span>        RWO            nfs</span>
<span>-</span>
<span>storage    </span>
<span>7d23h</span>
    5. 

  5. <span>[</span>
<span>root@k8s</span>
<span>-</span>
<span>02</span>
<span> zookeeper</span>
<span>]#</span>

检查Zookeeper集群状态

  1. <span>[</span>
<span>root@k8s</span>
<span>-</span>
<span>02</span>
<span> zookeeper</span>
<span>]#</span>
<span> kubectl </span>
<span>exec</span>
<span> </span>
<span>-</span>
<span>it </span>
<span>-</span>
<span>n kafka zookeeper</span>
<span>-</span>
<span>0</span>
<span> </span>
<span>--</span>
<span> bash</span>
    2. 

  2. <span>I have </span>
<span>no</span>
<span> name</span>
<span>!</span>
<span>@zookeeper</span>
<span>-</span>
<span>0</span>
<span>:/</span>
<span>$</span>
    3. 

  3. <span>I have </span>
<span>no</span>
<span> name</span>
<span>!</span>
<span>@zookeeper</span>
<span>-</span>
<span>0</span>
<span>:/</span>
<span>$ zkServer</span>
<span>.</span>
<span>sh status</span>
    4. 

  4. <span>/</span>
<span>opt</span>
<span>/</span>
<span>bitnami</span>
<span>/</span>
<span>java</span>
<span>/</span>
<span>bin</span>
<span>/</span>
<span>java</span>
    5. 

  5. <span>ZooKeeper</span>
<span> JMX enabled </span>
<span>by</span>
<span> </span>
<span>default</span>
    6. 

  6. <span>Using</span>
<span> config</span>
<span>:</span>
<span> </span>
<span>/opt/</span>
<span>bitnami</span>
<span>/</span>
<span>zookeeper</span>
<span>/</span>
<span>bin</span>
<span>/../</span>
<span>conf</span>
<span>/</span>
<span>zoo</span>
<span>.</span>
<span>cfg</span>
    7. 

  7. <span>Client</span>
<span> port found</span>
<span>:</span>
<span> </span>
<span>2181.</span>
<span> </span>
<span>Client</span>
<span> address</span>
<span>:</span>
<span> localhost</span>
<span>.</span>
<span> </span>
<span>Client</span>
<span> SSL</span>
<span>:</span>
<span> </span>
<span>false</span>
<span>.</span>
    8. 

  8. <span>Mode</span>
<span>:</span>
<span> follower</span>
    9. 

  9. <span>I have </span>
<span>no</span>
<span> name</span>
<span>!</span>
<span>@zookeeper</span>
<span>-</span>
<span>0</span>
<span>:/</span>
<span>$</span>

Helm 部署Kafka集群

拉取kafka

  1. <span>helm pull bitnami</span>
<span>/</span>
<span>kafka</span>

解压kafka

  1. <span>[</span>
<span>root@k8s</span>
<span>-</span>
<span>02</span>
<span> </span>
<span>~]#</span>
<span> tar xf kafka</span>
<span>-</span>
<span>22.1</span>
<span>.</span>
<span>3.tgz</span>

进入Kafka目录

  1. <span>[</span>
<span>root@k8s</span>
<span>-</span>
<span>02</span>
<span> </span>
<span>~]#</span>
<span> cd kafka</span>
    2. 

  2. <span>[</span>
<span>root@k8s</span>
<span>-</span>
<span>02</span>
<span> kafka</span>
<span>]#</span>
<span> ls</span>
    3. 

  3. <span>Chart</span>
<span>.</span>
<span>lock</span>
<span>  charts  </span>
<span>Chart</span>
<span>.</span>
<span>yaml  README</span>
<span>.</span>
<span>md  templates  values</span>
<span>.</span>
<span>yaml</span>

修改values.yaml

  1. <span>extraEnvVars</span>
<span>:</span>
<span> </span>
    2. 

  2. <span>  </span>
<span>-</span>
<span> name</span>
<span>:</span>
<span> TZ</span>
    3. 

  3. <span>    value</span>
<span>:</span>
<span> </span>
<span>"Asia/Shanghai"</span>
    4. 

  4. <span>---</span>
    5. 

    6. 

  6. <span># 副本数</span>
    7. 

  7. <span>replicaCount</span>
<span>:</span>
<span> </span>
<span>3</span>
<span>                    </span>
<span># 副本数</span>
    8. 

  8. <span>---</span>
    9. 

    10. 

  10. <span># 持久化存储</span>
    11. 

  11. <span>persistence</span>
<span>:</span>
    12. 

  12. <span>  enabled</span>
<span>:</span>
<span> </span>
<span>true</span>
    13. 

  13. <span>  storageClass</span>
<span>:</span>
<span> </span>
<span>"rook-ceph-block"</span>
<span>  </span>
<span># sc 有默认sc可以不写</span>
    14. 

  14. <span>  accessModes</span>
<span>:</span>
    15. 

  15. <span>    </span>
<span>-</span>
<span> </span>
<span>ReadWriteOnce</span>
    16. 

  16. <span>  size</span>
<span>:</span>
<span> </span>
<span>8Gi</span>
    17. 

  17. <span>---</span>
    18. 

  18. <span>kraft</span>
<span>:</span>
    19. 

  19. <span>  </span>
<span>## @param kraft.enabled Switch to enable or disable the Kraft mode for Kafka</span>
    20. 

  20. <span>  </span>
<span>##</span>
    21. 

  21. <span>  enabled</span>
<span>:</span>
<span> </span>
<span>false</span>
<span>   </span>
<span>#设置为false</span>
    22. 

  22. <span>---</span>
    23. 

  23. <span># 配置zookeeper外部连接</span>
    24. 

  24. <span>zookeeper</span>
<span>:</span>
    25. 

  25. <span>  enabled</span>
<span>:</span>
<span> </span>
<span>false</span>
<span>                   </span>
<span># 不使用内部zookeeper，默认是false</span>
    26. 

    27. 

  27. <span>externalZookeeper</span>
<span>:</span>
<span>                 </span>
<span># 外部zookeeper</span>
    28. 

  28. <span>  servers</span>
<span>:</span>
<span> zookeeper            </span>
<span>#Zookeeper svc名称</span>

可选配置

  1. <span>## 允许删除topic（按需开启）</span>
    2. 

  2. <span>deleteTopicEnable</span>
<span>:</span>
<span> </span>
<span>true</span>
    3. 

    4. 

  4. <span>## 日志保留时间（默认一周）</span>
    5. 

  5. <span>logRetentionHours</span>
<span>:</span>
<span> </span>
<span>168</span>
    6. 

    7. 

  7. <span>## 自动创建topic时的默认副本数</span>
    8. 

  8. <span>defaultReplicationFactor</span>
<span>:</span>
<span> </span>
<span>2</span>
    9. 

    10. 

  10. <span>## 用于配置offset记录的topic的partition的副本个数</span>
    11. 

  11. <span>offsetsTopicReplicationFactor</span>
<span>:</span>
<span> </span>
<span>2</span>
    12. 

    13. 

  13. <span>## 事务主题的复制因子</span>
    14. 

  14. <span>transactionStateLogReplicationFactor</span>
<span>:</span>
<span> </span>
<span>2</span>
    15. 

    16. 

  16. <span>## min.insync.replicas</span>
    17. 

  17. <span>transactionStateLogMinIsr</span>
<span>:</span>
<span> </span>
<span>2</span>
    18. 

    19. 

  19. <span>## 新建Topic时默认的分区数</span>
    20. 

  20. <span>numPartitions</span>
<span>:</span>
<span> </span>
<span>3</span>

创建Kafka集群

  1. <span>[</span>
<span>root@k8s</span>
<span>-</span>
<span>02</span>
<span> kafka</span>
<span>]#</span>
<span> helm install kafka </span>
<span>-</span>
<span>n kafka </span>
<span>.</span>
    2. 

    3. 

  3. <span>#输出结果如下</span>
    4. 

  4. <span>[</span>
<span>root@k8s</span>
<span>-</span>
<span>02</span>
<span> kafka</span>
<span>]#</span>
<span> helm install kafka </span>
<span>-</span>
<span>n kafka </span>
<span>.</span>
    5. 

  5. <span>W0523 </span>
<span>13</span>
<span>:</span>
<span>52</span>
<span>:</span>
<span>58.673090</span>
<span>   </span>
<span>28827</span>
<span> warnings</span>
<span>.</span>
<span>go</span>
<span>:</span>
<span>70</span>
<span>]</span>
<span> spec</span>
<span>.</span>
<span>template</span>
<span>.</span>
<span>spec</span>
<span>.</span>
<span>containers</span>
<span>[</span>
<span>0</span>
<span>].</span>
<span>env</span>
<span>[</span>
<span>39</span>
<span>].</span>
<span>name</span>
<span>:</span>
<span> duplicate name </span>
<span>"KAFKA_ENABLE_KRAFT"</span>
    6. 

  6. <span>NAME</span>
<span>:</span>
<span> kafka</span>
    7. 

  7. <span>LAST DEPLOYED</span>
<span>:</span>
<span> </span>
<span>Tue</span>
<span> </span>
<span>May</span>
<span> </span>
<span>23</span>
<span> </span>
<span>13</span>
<span>:</span>
<span>52</span>
<span>:</span>
<span>58</span>
<span> </span>
<span>2023</span>
    8. 

  8. <span>NAMESPACE</span>
<span>:</span>
<span> kafka</span>
    9. 

  9. <span>STATUS</span>
<span>:</span>
<span> deployed</span>
    10. 

  10. <span>REVISION</span>
<span>:</span>
<span> </span>
<span>1</span>
    11. 

  11. <span>TEST SUITE</span>
<span>:</span>
<span> </span>
<span>None</span>
    12. 

  12. <span>NOTES</span>
<span>:</span>
    13. 

  13. <span>CHART NAME</span>
<span>:</span>
<span> kafka</span>
    14. 

  14. <span>CHART VERSION</span>
<span>:</span>
<span> </span>
<span>22.1</span>
<span>.</span>
<span>3</span>
    15. 

  15. <span>APP VERSION</span>
<span>:</span>
<span> </span>
<span>3.4</span>
<span>.</span>
<span>0</span>
    16. 

    17. 

  17. <span>**</span>
<span> </span>
<span>Please</span>
<span> be patient </span>
<span>while</span>
<span> the chart </span>
<span>is</span>
<span> being deployed </span>
<span>**</span>
    18. 

    19. 

  19. <span>Kafka</span>
<span> can be accessed </span>
<span>by</span>
<span> consumers via port </span>
<span>9092</span>
<span> on the following DNS name </span>
<span>from</span>
<span> within your cluster</span>
<span>:</span>
    20. 

    21. 

  21. <span>    kafka</span>
<span>.</span>
<span>kafka</span>
<span>.</span>
<span>svc</span>
<span>.</span>
<span>cluster</span>
<span>.</span>
<span>local</span>
    22. 

    23. 

  23. <span>Each</span>
<span> </span>
<span>Kafka</span>
<span> broker can be accessed </span>
<span>by</span>
<span> producers via port </span>
<span>9092</span>
<span> on the following DNS name</span>
<span>(</span>
<span>s</span>
<span>)</span>
<span> </span>
<span>from</span>
<span> within your cluster</span>
<span>:</span>
    24. 

    25. 

  25. <span>    kafka</span>
<span>-</span>
<span>0.kafka</span>
<span>-</span>
<span>headless</span>
<span>.</span>
<span>kafka</span>
<span>.</span>
<span>svc</span>
<span>.</span>
<span>cluster</span>
<span>.</span>
<span>local</span>
<span>:</span>
<span>9092</span>
    26. 

  26. <span>    kafka</span>
<span>-</span>
<span>1.kafka</span>
<span>-</span>
<span>headless</span>
<span>.</span>
<span>kafka</span>
<span>.</span>
<span>svc</span>
<span>.</span>
<span>cluster</span>
<span>.</span>
<span>local</span>
<span>:</span>
<span>9092</span>
    27. 

  27. <span>    kafka</span>
<span>-</span>
<span>2.kafka</span>
<span>-</span>
<span>headless</span>
<span>.</span>
<span>kafka</span>
<span>.</span>
<span>svc</span>
<span>.</span>
<span>cluster</span>
<span>.</span>
<span>local</span>
<span>:</span>
<span>9092</span>
    28. 

    29. 

  29. <span>To</span>
<span> create a pod that you can </span>
<span>use</span>
<span> </span>
<span>as</span>
<span> a </span>
<span>Kafka</span>
<span> client run the following commands</span>
<span>:</span>
    30. 

    31. 

  31. <span>    kubectl run kafka</span>
<span>-</span>
<span>client </span>
<span>--</span>
<span>restart</span>
<span>=</span>
<span>'Never'</span>
<span> </span>
<span>--</span>
<span>image docker</span>
<span>.</span>
<span>io</span>
<span>/</span>
<span>bitnami</span>
<span>/</span>
<span>kafka</span>
<span>:</span>
<span>3.4</span>
<span>.</span>
<span>0</span>
<span>-</span>
<span>debian</span>
<span>-</span>
<span>11</span>
<span>-</span>
<span>r33 </span>
<span>--</span>
<span>namespace</span>
<span> kafka </span>
<span>--</span>
<span>command </span>
<span>--</span>
<span> sleep infinity</span>
    32. 

  32. <span>    kubectl </span>
<span>exec</span>
<span> </span>
<span>--</span>
<span>tty </span>
<span>-</span>
<span>i kafka</span>
<span>-</span>
<span>client </span>
<span>--</span>
<span>namespace</span>
<span> kafka </span>
<span>--</span>
<span> bash</span>
    33. 

    34. 

  34. <span>    PRODUCER</span>
<span>:</span>
    35. 

  35. <span>        kafka</span>
<span>-</span>
<span>console</span>
<span>-</span>
<span>producer</span>
<span>.</span>
<span>sh \</span>
    36. 

  36. <span>            </span>
<span>--</span>
<span>broker</span>
<span>-</span>
<span>list kafka</span>
<span>-</span>
<span>0.kafka</span>
<span>-</span>
<span>headless</span>
<span>.</span>
<span>kafka</span>
<span>.</span>
<span>svc</span>
<span>.</span>
<span>cluster</span>
<span>.</span>
<span>local</span>
<span>:</span>
<span>9092</span>
<span>,</span>
<span>kafka</span>
<span>-</span>
<span>1.kafka</span>
<span>-</span>
<span>headless</span>
<span>.</span>
<span>kafka</span>
<span>.</span>
<span>svc</span>
<span>.</span>
<span>cluster</span>
<span>.</span>
<span>local</span>
<span>:</span>
<span>9092</span>
<span>,</span>
<span>kafka</span>
<span>-</span>
<span>2.kafka</span>
<span>-</span>
<span>headless</span>
<span>.</span>
<span>kafka</span>
<span>.</span>
<span>svc</span>
<span>.</span>
<span>cluster</span>
<span>.</span>
<span>local</span>
<span>:</span>
<span>9092</span>
<span> \</span>
    37. 

  37. <span>            </span>
<span>--</span>
<span>topic test</span>
    38. 

    39. 

  39. <span>    CONSUMER</span>
<span>:</span>
    40. 

  40. <span>        kafka</span>
<span>-</span>
<span>console</span>
<span>-</span>
<span>consumer</span>
<span>.</span>
<span>sh \</span>
    41. 

  41. <span>            </span>
<span>--</span>
<span>bootstrap</span>
<span>-</span>
<span>server kafka</span>
<span>.</span>
<span>kafka</span>
<span>.</span>
<span>svc</span>
<span>.</span>
<span>cluster</span>
<span>.</span>
<span>local</span>
<span>:</span>
<span>9092</span>
<span> \</span>
    42. 

  42. <span>            </span>
<span>--</span>
<span>topic test \</span>
    43. 

  43. <span>            </span>
<span>--</span>
<span>from</span>
<span>-</span>
<span>beginning</span>

进入Kafka集群，创建topic查看

  1. <span>##进入Kafka集群</span>
    2. 

  2. <span>kubectl </span>
<span>exec</span>
<span> </span>
<span>-</span>
<span>it </span>
<span>-</span>
<span>n kafka kafka</span>
<span>-</span>
<span>0</span>
<span> </span>
<span>--</span>
<span> bash</span>
    3. 

    4. 

  4. <span>#创建topic</span>
    5. 

  5. <span>kafka</span>
<span>-</span>
<span>topics</span>
<span>.</span>
<span>sh </span>
<span>--</span>
<span>create </span>
<span>--</span>
<span>bootstrap</span>
<span>-</span>
<span>server kafka</span>
<span>:</span>
<span>9092</span>
<span>  </span>
<span>--</span>
<span>topic abcdocker</span>
    6. 

    7. 

  7. <span>#查看topic列表</span>
    8. 

  8. <span>kafka</span>
<span>-</span>
<span>topics</span>
<span>.</span>
<span>sh </span>
<span>--</span>
<span>list </span>
<span>--</span>
<span>bootstrap</span>
<span>-</span>
<span>server kafka</span>
<span>:</span>
<span>9092</span>
<span> </span>
    9. 

    10. 

  10. <span>#查看topic详细信息</span>
    11. 

  11. <span>kafka</span>
<span>-</span>
<span>topics</span>
<span>.</span>
<span>sh </span>
<span>--</span>
<span>bootstrap</span>
<span>-</span>
<span>server kafka</span>
<span>:</span>
<span>9092</span>
<span>  </span>
<span>--</span>
<span>describe </span>
<span>--</span>
<span>topic abcdocker</span>
    12. 

    13. 

  13. <span>#配置文件配置已经生效，默认分区为3，副本为3，过期时间为168小时</span>
    14. 

    15. 

  15. <span>I have </span>
<span>no</span>
<span> name</span>
<span>!</span>
<span>@kafka</span>
<span>-</span>
<span>0</span>
<span>:/</span>
<span>$ kafka</span>
<span>-</span>
<span>topics</span>
<span>.</span>
<span>sh </span>
<span>--</span>
<span>bootstrap</span>
<span>-</span>
<span>server kafka</span>
<span>:</span>
<span>9092</span>
<span>  </span>
<span>--</span>
<span>describe </span>
<span>--</span>
<span>topic abcdocker</span>
    16. 

  16. <span>Topic</span>
<span>:</span>
<span> abcdocker    </span>
<span>TopicId</span>
<span>:</span>
<span> jcJtxY1NSr</span>
<span>-</span>
<span>nSloax8oPnA </span>
<span>PartitionCount</span>
<span>:</span>
<span> </span>
<span>3</span>
<span>   </span>
<span>ReplicationFactor</span>
<span>:</span>
<span> </span>
<span>3</span>
<span>    </span>
<span>Configs</span>
<span>:</span>
<span> flush</span>
<span>.</span>
<span>ms</span>
<span>=</span>
<span>1000</span>
<span>,</span>
<span>segment</span>
<span>.</span>
<span>bytes</span>
<span>=</span>
<span>1073741824</span>
<span>,</span>
<span>flush</span>
<span>.</span>
<span>messages</span>
<span>=</span>
<span>10000</span>
<span>,</span>
<span>max</span>
<span>.</span>
<span>message</span>
<span>.</span>
<span>bytes</span>
<span>=</span>
<span>1000012</span>
<span>,</span>
<span>retention</span>
<span>.</span>
<span>bytes</span>
<span>=</span>
<span>1073741824</span>
    17. 

  17. <span>    </span>
<span>Topic</span>
<span>:</span>
<span> abcdocker    </span>
<span>Partition</span>
<span>:</span>
<span> </span>
<span>0</span>
<span>    </span>
<span>Leader</span>
<span>:</span>
<span> </span>
<span>1</span>
<span>   </span>
<span>Replicas</span>
<span>:</span>
<span> </span>
<span>1</span>
<span>,</span>
<span>2</span>
<span>,</span>
<span>0</span>
<span> </span>
<span>Isr</span>
<span>:</span>
<span> </span>
<span>1</span>
<span>,</span>
<span>2</span>
<span>,</span>
<span>0</span>
    18. 

  18. <span>    </span>
<span>Topic</span>
<span>:</span>
<span> abcdocker    </span>
<span>Partition</span>
<span>:</span>
<span> </span>
<span>1</span>
<span>    </span>
<span>Leader</span>
<span>:</span>
<span> </span>
<span>0</span>
<span>   </span>
<span>Replicas</span>
<span>:</span>
<span> </span>
<span>0</span>
<span>,</span>
<span>1</span>
<span>,</span>
<span>2</span>
<span> </span>
<span>Isr</span>
<span>:</span>
<span> </span>
<span>0</span>
<span>,</span>
<span>1</span>
<span>,</span>
<span>2</span>
    19. 

  19. <span>    </span>
<span>Topic</span>
<span>:</span>
<span> abcdocker    </span>
<span>Partition</span>
<span>:</span>
<span> </span>
<span>2</span>
<span>    </span>
<span>Leader</span>
<span>:</span>
<span> </span>
<span>2</span>
<span>   </span>
<span>Replicas</span>
<span>:</span>
<span> </span>
<span>2</span>
<span>,</span>
<span>0</span>
<span>,</span>
<span>1</span>
<span> </span>
<span>Isr</span>
<span>:</span>
<span> </span>
<span>2</span>
<span>,</span>
<span>0</span>
<span>,</span>
<span>1</span>