kubeprometheus监控集群外部主机

简介

k8s使用的是kube-prometheus监控的

操作

网上有的人是创建endpoint，然后使用serviceMonitor去监控的，也就是说如果我有100台主机，那么就要创建100个serviceMonitor就是例如下面这样

---
apiVersion: v1
kind: Service
metadata:
  name: ceph-k8s
  labels:
    component: ceph-k8s
spec:
  type: ExternalName
  externalName: {EXTERNAL_IP}
  ports:
  - port: 9283
    name: scrape
    protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
  name: ceph-k8s
  labels:
    k8s-app: ceph-k8s
subsets:
  - addresses:
    - ip: {EXTERNAL_IP}
    ports:
      - port: 9283
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ceph-k8s
  labels:
    k8s-app: ceph-k8s
  spec:
    endpoints:
      - port: scrape
        interval: 30s
        path: /
      selector:
        matchLabels:
          component: ceph-k8s
      namespaceSelector:
        matchNames:
        - kube-system

这显然不是最佳做法，百度了一下发现prometheus可以添加额外的配置文件，详细的可以看下面

https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/additional-scrape-config.md

那么思路就有了，在additional-scrape-config里面添加job，然后使用file_sd去发现主机就可以了，而file直接创建成configmap挂载到容器里面，之后添加新的监控主机的话直接修改configmap就可以了

添加配置

prometheus-additional.yaml

- job_name: "vminsert"
  static_configs:
  - targets: ["192.168.26.176:8480"]

创建secret

kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml
kubectl apply -f additional-scrape-configs.yaml -n monitoring

最后，在 CRD 中引用此附加配置。prometheus.yaml

  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 2.32.1
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - apiVersion: v2
      name: alertmanager-main
      namespace: monitoring
      port: web
  enableFeatures: []
  externalLabels: {}
  image: quay.io/prometheus/prometheus:v2.32.1
  nodeSelector:
    kubernetes.io/os: linux
  podMetadata:
    labels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
      app.kubernetes.io/version: 2.32.1
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  probeNamespaceSelector: {}
  probeSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  remoteWrite:
  - url: http://192.168.26.176:8480/insert/0/prometheus/
    writeRelabelConfigs:
    - action: labeldrop
      regex: prometheus_replica
  ruleNamespaceSelector: {}
  ruleSelector: {}
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: 2.32.1
  retention: 12h
  enableAdminAPI: true
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml