Prometheus Alertmanager告警持久化_统计告警次数-每日运维

下午突发奇想，天天企业微信发送告警，但是没有统计过告警都收到过哪些，都是哪些级别，持续的时间又是多长，下午在官方文档找到一个项目alertsnitch。可以对我们的Alertmanager发送的告警进行持久化，实际上就是一个webhook接收器，Alertmanager的数据采集同步写入到Mysql中为以后的数据分析

效果图下

由于官方Alertmanager 本身不支持的通知机制，所以我们需要webhook 接收器进行集成，将数据发送给后端Mysql或者是Postgres
官方gitlab地址:https://gitlab.com/yakshaving.art/alertsnitch

数据我这里存储在MySQL中，我们需要安装一套Mysql配置
创建持久化目录

mkdir -p /data/mysql/{conf,data,logs}

创建自定义my.cnf

cat >/data/mysql/conf/my.cnf<<EOF
[mysqld]
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
datadir         = /var/lib/mysql
secure-file-priv= NULL
lower_case_table_names = 1
EOF

lower_case_table_names需要开启，不区分大小写
启动mysql容器

docker run -p 3306:3306 --name mysql 
  -v /data/mysql/conf:/etc/mysql/conf.d 
  -v /data/mysql/logs:/logs 
  -v /data/mysql/data:/var/lib/mysql 
  -e MYSQL_ROOT_PASSWORD=abcdocker -d 
  --restart=always mysql

#这里我临时安装一套mysql，可以根据实际场景自行部署

创建用户

[root@prometheus alert]# docker exec -it mysql bash
root@b7b076e1c2a9:/# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 8
Server version: 8.0.27 MySQL Community Server - GPL

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

mysql> create user 'i4t'@'%' identified by 'daskjdlajdas@3';
Query OK, 0 rows affected (0.02 sec)

mysql> grant all privileges on *.* to 'i4t'@'%' ;
Query OK, 0 rows affected (0.01 sec)

我们在创建个库，后续使用i4t 将数据写进库里

mysql> create database alert;
Query OK, 1 row affected (0.01 sec)

接下来在alert库中写入一些表结构数据

#进入到业务库中
USE alert

DROP PROCEDURE IF EXISTS bootstrap;

DELIMITER //
CREATE PROCEDURE bootstrap()
BEGIN
  SET @exists := (SELECT 1 FROM information_schema.tables I WHERE I.table_name = "Model" AND I.table_schema = database());
  IF @exists IS NULL THEN

    CREATE TABLE `Model` (
      `ID` enum('1') NOT NULL,
      `version` VARCHAR(20) NOT NULL,
      PRIMARY KEY (`ID`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8;

    INSERT INTO `Model` (`version`) VALUES ("0.0.1");

  ELSE
    SIGNAL SQLSTATE '42000' SET MESSAGE_TEXT='Model Table Exists, quitting...';
  END IF;
END;
//
DELIMITER ;

-- Execute the procedure
CALL bootstrap();

-- Drop the procedure
DROP PROCEDURE bootstrap;

-- Create the rest of the tables
CREATE TABLE `AlertGroup` (
    `ID` INT NOT NULL AUTO_INCREMENT,
    `time` TIMESTAMP NOT NULL,
    `receiver` VARCHAR(100) NOT NULL,
    `status` VARCHAR(50) NOT NULL,
    `externalURL` TEXT NOT NULL,
    `groupKey` VARCHAR(255) NOT NULL,
    KEY `idx_time` (`time`) USING BTREE,
    KEY `idx_status_ts` (`status`, `time`) USING BTREE,
    PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE `GroupLabel` (
    `ID` INT NOT NULL AUTO_INCREMENT,
    `AlertGroupID` INT NOT NULL,
    `GroupLabel` VARCHAR(100) NOT NULL,
    `Value` VARCHAR(1000) NOT NULL,
    FOREIGN KEY (AlertGroupID) REFERENCES AlertGroup (ID) ON DELETE CASCADE,
    PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE `CommonLabel` (
    `ID` INT NOT NULL AUTO_INCREMENT,
    `AlertGroupID` INT NOT NULL,
    `Label` VARCHAR(100) NOT NULL,
    `Value` VARCHAR(1000) NOT NULL,
    FOREIGN KEY (AlertGroupID) REFERENCES AlertGroup (ID) ON DELETE CASCADE,
    PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE `CommonAnnotation` (
    `ID` INT NOT NULL AUTO_INCREMENT,
    `AlertGroupID` INT NOT NULL,
    `Annotation` VARCHAR(100) NOT NULL,
    `Value` VARCHAR(1000) NOT NULL,
    FOREIGN KEY (AlertGroupID) REFERENCES AlertGroup (ID) ON DELETE CASCADE,
    PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE `Alert` (
    `ID` INT NOT NULL AUTO_INCREMENT,
    `alertGroupID` INT NOT NULL,
    `status` VARCHAR(50) NOT NULL,
    `startsAt` DATETIME NOT NULL,
    `endsAt` DATETIME DEFAULT NULL,
    `generatorURL` TEXT NOT NULL,
    FOREIGN KEY (alertGroupID) REFERENCES AlertGroup (ID) ON DELETE CASCADE,
    PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE `AlertLabel` (
    `ID` INT NOT NULL AUTO_INCREMENT,
    `AlertID` INT NOT NULL,
    `Label` VARCHAR(100) NOT NULL,
    `Value` VARCHAR(1000) NOT NULL,
    FOREIGN KEY (AlertID) REFERENCES Alert (ID) ON DELETE CASCADE,
    PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE `AlertAnnotation` (
    `ID` INT NOT NULL AUTO_INCREMENT,
    `AlertID` INT NOT NULL,
    `Annotation` VARCHAR(100) NOT NULL,
    `Value` VARCHAR(1000) NOT NULL,
    FOREIGN KEY (AlertID) REFERENCES Alert (ID) ON DELETE CASCADE,
    PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

在修改Model源版本

ALTER TABLE Alert 
    ADD `fingerprint` TEXT NOT NULL
;

UPDATE `Model`  SET `version`="0.1.0";

启动项目

#首先设置环境变量
export ALERTSNITCH_BACKEND="mysql"
export ALERTSNITCH_DSN=DB_USER:DB_PASSWORD@(DB_IP:DB_PORT)/DB_NAME"

#根据实际情况修改

启动Docker 容器

docker run -itd 
    -p 9567:9567 
    --name alertsnitch 
    -e ALERTSNITCH_DSN 
    -e ALERTSNITCH_BACKEND 
    registry.gitlab.com/yakshaving.art/alertsnitch

接下来配置Alertmanager

#receivers告警信息如下
receivers:
- name: alertsnitch
  webhook_configs:
    - url: http://<alertsnitch-host-or-ip>:9567/webhook

Alert route如下

route:
  routes:
  - receiver: alertsnitch
    continue: true

完整配置Alertmanager告警配置如下

[root@prometheus alertmanager]# cat config.yml
global:
  resolve_timeout: 5m
  http_config:
    follow_redirects: true
  smtp_hello: localhost
  smtp_require_tls: true
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
  opsgenie_api_url: 'https://api.opsgenie.com/'
  wechat_api_url: 'https://qyapi.weixin.qq.com/cgi-bin/ '
  wechat_api_corp_id: wwxxxx #企业id
  victorops_api_url: 'https://alert.victorops.com/integrations/generic/20131114/alert/'
route:
  receiver: "null"
  group_wait: 30s 
  group_interval: 3m
  repeat_interval: 3m
  routes:
  - receiver: abcdocker
    continue: true
  - receiver: alertsnitch
    continue: false

receivers:
- name: abcdocker
  wechat_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
    api_secret: xxxW-rsXEtY_oFTvLk # 申请企业微信应用后生成的密码
    corp_id: wwcxxx50d
    message: '{{ template "wechat.default.message" . }}'
    api_url: https://qyapi.weixin.qq.com/cgi-bin/
    to_user: abcdocker #发送到某一用户也可以 @all 就是群组全员发送
    to_party: '{{ template "wechat.default.to_party" . }}'
    to_tag: '{{ template "wechat.default.to_tag" . }}'
    agent_id: "100004" #申请企业微信应用id
    message_type: text
- name: "null"
- name: alertsnitch
  webhook_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
    url: http://10.0.24.13:9567/webhook
    max_alerts: 0
templates:
- /etc/alertmanager/template/*.tmpl #告警模板路径

重启Alertmanager服务

[root@prometheus alertmanager]# docker restart 9780c1d1d924
9780c1d1d924

Alertmanager 启动成功后，我们需要去Grafana配置Mysql数据源

因为数据都存储在Mysql中，所以我们grafana的数据源来自Mysql

搜索Mysql数据源

根据刚刚我们创建的用户信息填写

检测是否正常通信

接下来导入模板，ID15833

导入即可

在Mysql中已经可以看到日志了
Grafana 效果图如下

Prometheus 监控VMware_ESXI并配置AlertManager告警
Prometheus Grafana使用Ceph持久化并监控k8s集群
Prometheus Blackbox域名SSL证书监控并设置AlertManager告警
Prometheus监控Ceph集群并设置AlertManager告警

Prometheus Alertmanager告警持久化_统计告警次数

效果图下

相关文章：

相关文章

发布评论取消回复

泡泡

百度搜索：蓝易云 – Docker的安装与卸载(Centos、Ubuntu )

PostgreSQL系数据库使用COPY导数时如何实现增量及重复数据更新导入

【OceanBase DBA早下班系列】—— obdiag 收集的 SQL Monitor Report 如何解读

Prometheus Alertmanager告警持久化_统计告警次数

效果图下

相关文章：

相关文章

发布评论 取消回复

泡泡

百度搜索：蓝易云 – Docker的安装与卸载(Centos、Ubuntu )

PostgreSQL系数据库使用COPY导数时如何实现增量及重复数据更新导入

【OceanBase DBA早下班系列】—— obdiag 收集的 SQL Monitor Report 如何解读

发布评论取消回复