nginx全方面监控Fluentd+Nginx Prometheus Exporter

安装 Nginx

安装nginx

yum install -y nginx

确保基本http_stub_status_module模块配置了nginx

[root@vms77 ~]# nginx -V
nginx version: nginx/1.20.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
built with OpenSSL 1.1.1k  FIPS 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'

启动nginx

sudo systemctl start nginx

查看nginx运行状态

sudo systemctl status nginx

公开基本的 Nginx指标

让我们创建一个新的 Nginx 配置文件，以使用我们的status模块添加额外的模块。如果您使用不同的方法来安装 Nginx则 Nginx 配置的位置可能不同。

现在是配置文件。

vim /etc/nginx/conf.d/status.conf

（可选）您可以限制此插件仅向本地主机发出指标。如果您有一个 Nginx 实例并且还在其上安装了 Prometheus 导出器，这可能会很有用。如果您有多个 Nginx 服务器，最好将 prometheus 导出器部署在单独的实例上，并从单个导出器中抓取所有服务器。

我们将使用 Nginx 指令在端口 8080 页面上公开基本指标。location/status

server {
 listen 8080; 
 # Optionally: allow access only from localhost 
 # listen 127.0.0.1:8080; 
 server_name _; 
 location /status { 
   stub_status; 
  } 
}

在重新启动 Nginx 之前，请始终验证配置是否有效。

nginx -t

重启Nginx

systemctl reload nginx

现在我们可以访问页面了。http://:8080/status

[root@vms77 ~]# curl http://127.0.0.1:8080/status
Active connections: 1
server accepts handled requests
 2 2 2
Reading: 0 Writing: 1 Waiting: 0

但是这些指标不太全需要自己去完善

安装 Nginx Prometheus Exporter

不过，现在让我们获取所有可用的指标。我们将使用 Nginx prometheus 导出器来做到这一点。这是一个 Golang 应用程序，可以编译为单个二进制文件，无需外部依赖项，非常易于安装。

创建文件

mkdir /opt/nginx-exporter
cd /opt/nginx-exporter

创建用户和组nginx-exporter

sudo useradd --system --no-create-home --shell /bin/false nginx-exporter

我们可以使用 curl 下载

curl -L https://github.com/nginxinc/nginx-prometheus-exporter/releases/download/v0.11.0/nginx-prometheus-exporter_0.11.0_linux_amd64.tar.gz -o nginx-prometheus-exporter_0.11.0_linux_amd64.tar.gz

解压

tar -zxf nginx-prometheus-exporter_0.11.0_linux_amd64.tar.gz

检查版本

./nginx-prometheus-exporter --version

赋权

chown -R nginx-exporter:nginx-exporter /opt/nginx-exporter

systemd 管理

vim /etc/systemd/system/nginx-exporter.service
[Unit]
Description=Nginx Exporter
Wants=network-online.target
After=network-online.target

StartLimitIntervalSec=0

[Service]
User=nginx-exporter
Group=nginx-exporter
Type=simple
Restart=on-failure
RestartSec=5s

ExecStart=/opt/nginx-exporter/nginx-prometheus-exporter 
    -nginx.scrape-uri=http://localhost:8080/status

[Install]
WantedBy=multi-user.target

开机自启

systemctl enable nginx-exporter

启动

systemctl start nginx-exporter

检查服务的状态。

systemctl status nginx-exporter

如果nginx-exporter无法启动，您可以检查日志以查找错误消息。

journalctl -u nginx-exporter -f --no-pager

要验证 Prometheus 导出器是否可以访问 nginx 并正确抓取指标，请使用导出器的 curl 命令和默认端口。9113

curl localhost:9113/metrics

现在，您应该能够从状态页面获取相同的指标，但采用普罗米修斯格式。

[root@vms77 nginx-exporter]# curl localhost:9113/metrics
# HELP nginx_connections_accepted Accepted client connections
# TYPE nginx_connections_accepted counter
nginx_connections_accepted 4
# HELP nginx_connections_active Active client connections
# TYPE nginx_connections_active gauge
nginx_connections_active 1
# HELP nginx_connections_handled Handled client connections
# TYPE nginx_connections_handled counter
nginx_connections_handled 4
# HELP nginx_connections_reading Connections where NGINX is reading the request header
# TYPE nginx_connections_reading gauge
nginx_connections_reading 0
# HELP nginx_connections_waiting Idle client connections
# TYPE nginx_connections_waiting gauge
nginx_connections_waiting 0
# HELP nginx_connections_writing Connections where NGINX is writing the response back to the client
# TYPE nginx_connections_writing gauge
nginx_connections_writing 1
# HELP nginx_http_requests_total Total http requests
# TYPE nginx_http_requests_total counter
nginx_http_requests_total 4
# HELP nginx_up Status of the last metric scrape
# TYPE nginx_up gauge
nginx_up 1
# HELP nginxexporter_build_info Exporter build information
# TYPE nginxexporter_build_info gauge
nginxexporter_build_info{arch="linux/amd64",commit="e4a6810d4f0b776f7fde37fea1d84e4c7284b72a",date="2022-09-07T21:09:51Z",dirty="false",go="go1.19",version="0.11.0"} 1

安装prometheus

为 Prometehus 创建一个专用的 用户。

sudo useradd --system --no-create-home --shell /bin/false prometheus

让我们从下载页面查看最新版本的普罗米修斯。

下载prometheus

curl -L https://github.com/prometheus/prometheus/releases/download/v2.41.0/prometheus-2.41.0.linux-amd64.tar.gz -o prometheus-2.41.0.linux-amd64.tar.gz

解压

tar -xvf prometheus-2.41.0.linux-amd64.tar.gz

创建文件

mkdir -p /data /etc/prometheus
cd prometheus-2.41.0.linux-amd64
mv prometheus promtool /usr/local/bin/
mv consoles/ console_libraries/ /etc/prometheus/
mv prometheus.yml /etc/prometheus/prometheus.yml

赋权 /etc/prometheus/ /data

chown -R prometheus:prometheus /etc/prometheus/ /data/

检查版本

prometheus --version

systemd 配置文件。

vim /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

StartLimitIntervalSec=0

[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/prometheus 
  --config.file=/etc/prometheus/prometheus.yml 
  --storage.tsdb.path=/data 
  --web.console.templates=/etc/prometheus/consoles 
  --web.console.libraries=/etc/prometheus/console_libraries 
  --web.listen-address=0.0.0.0:9090 
  --web.enable-lifecycle

[Install]
WantedBy=multi-user.target

重启之前，添加的Nginx Prometheus Exporter作为目标。

vim /etc/prometheus/prometheus.yml

添加配置

scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: "nginx-prometheus-exporter"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9113"]

添加开机自启

systemctl enable prometheus

启动

systemctl start prometheus

查看的状态

systemctl status prometheus

查看 http://:9090/

在Targets下，有一个nginx-prometheus-exporter

查询活动的 nginx 连接nginx_connections_active

安装Grafana

安装 Grafana

sudo yum install -y https://dl.grafana.com/oss/release/grafana-10.0.2-1.x86_64.rpm

加入开机自启

systemctl enable grafana-server

启动

systemctl start grafana-server

查看状态

systemctl status grafana-server

用户名为admin ，密码也为admin url: http://:3000

• 首先，让我们添加我们的普罗米修斯作为数据源。
• 对于 URL，请使用并单击保存并测试。http://localhost:9090
• 让我们创建一个新的仪表板并将其称为 。Nginx
• 创建新面板。
• 对于指标，请使用 .nginx_connections_active
• 对于传奇.{{ instance }}
• 标题：。Active Connections

我将使用我们从状态页面重试的指标填写其余面板。您可以在我的 github 存储库中找到此仪表板。

在左侧，我们有所有的仪表，在右侧，所有的计数器。要使用计数器，您需要应用速率函数，例如，计算每秒的请求数。

安装Fluentd

下一步是安装 Fluentd 并将 Nginx 日志转换为 Prometheus 指标。

首先，我们需要安装release仓库

sudo yum install -y centos-release-scl epel-release

安装 ruby 最低版本2.7

sudo yum install -y rh-ruby27 gcc-c++ make ruby-devel

编译

sudo yum install -y git gcc bzip2 openssl-devel libyaml-devel libffi-devel readline-devel zlib-devel gdbm-devel ncurses-devel
wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.0.tar.gz
tar -xzvf ruby-2.7.0.tar.gz
cd ruby-2.7.0
./configure
make
sudo make install

该步骤报错

sudo yum install -y rh-ruby27-ruby-devel

• 打开 /etc/ld.so.conf.d/rh-ruby27.conf 文件以编辑：

  sudo vi /etc/ld.so.conf.d/rh-ruby27.conf
  
  

• 在文件中添加以下行：

  /opt/rh/rh-ruby27/root/usr/lib64
  

• 保存文件并运行以下命令更新运行时库缓存：

  sudo ldconfig
  

验证版本

ruby --version

gem 安装 fluentd 以及使用国内镜像（使用https 正式整不明白就http安装）

gem install fluentd --no-document --source http://mirrors.tuna.tsinghua.edu.cn/rubygems

安装fluent-plugin-prometheus

gem install fluent-plugin-prometheus --source http://mirrors.tuna.tsinghua.edu.cn/rubygems

检查是否安装fluent-plugin-prometheus

检查最新版本。

fluentd --version

创建一个文件夹

mkdir /etc/fluent/

让我们更新 Nginx 访问日志以发出其他值。

vim /etc/nginx/nginx.conf

在日志末尾添加。但最好用我的替换整个字符串以避免任何拼写错误。upstream_response_time

    log_format custom '$remote_addr - $remote_user [$time_local] '
    '"$request" $status $body_bytes_sent '
    '"$http_referer" "$http_user_agent" '
    '$upstream_response_time';

    access_log /var/log/nginx/access.log custom;

保存并重新启动 Nginx。

nginx -t
systemctl restart nginx

现在我们需要创建一个正则表达式来解析 nginx 访问日志。首先，让我们从日志中获取一个示例并使用其中一个在线正则表达式编辑器。

tail -f /var/log/nginx/access.log

由于我们没有任何上游服务，因此最后一个值现在为空。

这里有旧日志;让我们调用其中一个 Nginx 端点。

127.0.0.1 - - [12/Jul/2023:22:08:58 +0800] "GET / HTTP/1.1" 200 4833 "-" "curl/7.29.0" -

让我们使用 regex101com 来创建正则表达式。

^(?[^ ]*) (?[^ ]*) (?[^ ]*) [(?[^]]*)] "(?w+)(?:s+(?[^"]*?)(?:s+S*)?)?" (?[^ ]*) (?[^ ]*)(?:s"(?[^"]*)") "(?[^"]*)" (?[^ ]*)$

粘贴日志示例并输入正则表达式，例如状态代码、方法、大小等。所有这些我们将能够转换为普罗米修斯并将它们用作标签。

创建 Fluentd 配置。

vim /etc/fluent/fluent.conf

您将找到我们将日志转换为普罗米修斯指标的指标部分。例如，为了测量延迟，我们使用nginx_upstream_time_seconds_hist

    @type prometheus_tail_monitor



  @type prometheus



    @type tail
    
    @type regexp
    expression /^(?[^ ]*) (?[^ ]*) (?[^ ]*) [(?[^]]*)] "(?w+)(?:s+(?[^"]*?)(?:s+S*)?)?" (?[^ ]*) (?[^ ]*)(?:s"(?[^"]*)") "(?[^"]*)" (?[^ ]*)$/
        time_format %d/%b/%Y:%H:%M:%S %z
        keep_time_key true
        types size:integer,reqtime:float,uct:float,uht:float,urt:float
    
    tag nginx
    path /var/log/nginx/access.log
    pos_file /tmp/fluent_nginx.pos


    @type prometheus

  
    name nginx_size_bytes_total
    type counter
    desc nginx bytes sent
    key size
  

  
    name nginx_request_status_code_total
    type counter
    desc nginx request status code
    
      method ${method}
      path ${path}
      status_code ${status_code}
    
  

  
    name nginx_http_request_duration_seconds
    type histogram
    desc Histogram of the total time spent on receiving the response from the upstream server.
    key urt
    
      method ${method}
      path ${path}
      status_code ${status_code}
    
  

fluentd systemd启动文件

vim /etc/systemd/system/fluentd.service



[Unit]
Description=Fluentd
Wants=network-online.target
After=network-online.target

StartLimitIntervalSec=0

[Service]
User=root
Group=root
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=ExecStart=/opt/rh/rh-ruby27/root/usr/local/bin/fluentd --config /etc/fluent/fluent.conf

[Install]
WantedBy=multi-user.target

与往常一样，启用，启动并检查状态。

systemctl enable fluentd
systemctl start fluentd
systemctl status fluentd

访问指标。localhost:24231/metrics

curl http://localhost:24231/metrics

让我们添加另一个普罗米修斯目标来流畅地抓取。

vim /etc/prometheus/prometheus.yml



  - job_name: "nginx-fluentd"
    static_configs:
      - targets: ["localhost:24231"]

重新启动普罗米修斯。

systemctl restart prometheus

现在，普罗米修斯UI中应该有2个目标。

创建go程序测试

创建一个简单的 go web应用来测试指标

package main

import (
	"fmt"
	"math/rand"
	"net/http"
	"time"
)

type Device struct {
	ID       int    `json:"id"`
	MAC      string `json:"mac"`
	Firmware string `json:"firmware"`
}

var devices = []Device{
	{
		ID:       1,
		MAC:      "14-BA-17-74-24-1D",
		Firmware: "2.0.6",
	},
	{
		ID:       2,
		MAC:      "14-BA-17-74-24-1D",
		Firmware: "2.0.6",
	},
}

func main() {
	rand.Seed(time.Now().UnixNano())

	http.HandleFunc("/api/devices", devicesHandler)

	fmt.Println("Server listening on port 8000...")
	http.ListenAndServe(":8000", nil)
}

func devicesHandler(w http.ResponseWriter, r *http.Request) {
	sleep()
	if r.Method == http.MethodPost {
		w.WriteHeader(http.StatusCreated)
		w.Write([]byte(`{"message": "Device created!"}`))
	} else {
		w.Header().Set("Content-Type", "application/json")
		w.WriteHeader(http.StatusOK)
		// Convert devices to JSON and write the response
		// You can use a JSON library like encoding/json to marshal the devices slice into JSON format
		// Here is a simple example without using external libraries:
		response := "["
		for i, device := range devices {
			response += fmt.Sprintf(`{"id": %d, "mac": "%s", "firmware": "%s"}`, device.ID, device.MAC, device.Firmware)
			if i != len(devices)-1 {
				response += ","
			}
		}
		response += "]"
		w.Write([]byte(response))
	}
}

func sleep() {
	time.Sleep(time.Duration(rand.Intn(6)) * 100 * time.Millisecond)
}

创建文件夹

mkdir -p /opt/myapp

编译

 go build  -o /opt/app/app test.go

创建用户

sudo useradd --system --no-create-home --shell /bin/false myapp

更新文件夹的所有权。

chown -R myapp:myapp /opt/myapp

为应用创建 systemd 服务。

vim /etc/systemd/system/myapp.service



[Unit]
Description=My App
Wants=network-online.target
After=network-online.target

StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
User=myapp
Group=myapp
Type=simple
Restart=on-failure
RestartSec=5s

ExecStart=/opt/app/app

[Install]
WantedBy=multi-user.target

启用、启动并检查应用程序。

systemctl enable myapp
systemctl start myapp
systemctl status myapp

现在，检查您是否可以在本地访问烧瓶应用程序。

curl -i localhost:8000/api/devices

下一步是使用 Nginx 作为我们应用程序的反向代理。

vim /etc/nginx/conf.d/myapp.conf

让我们也添加一些标头，以防烧瓶需要知道请求来自哪里。

server {
    listen 80;
    server_name _;

   location / {
        proxy_pass http://127.0.0.1:8000/api/devices;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Prefix /;
    }
}

删除默认的 Nginx 配置与 hello 页面。

rm /etc/nginx/conf.d/default.conf

测试并重新加载 Nginx 配置。

nginx -t
systemctl reload nginx

现在我们可以使用 Nginx IP 地址来测试应用程序。http://

最后，让我们再创建一些 Grafana 仪表板来使用这些指标下面是导出数据

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 2,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "ea60a2ad-a704-42cf-8bcd-9e2e2dd02366"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 50,
            "gradientMode": "opacity",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "smooth",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": true,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "decimals": 1,
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "s"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "P99"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "red",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "P90"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "yellow",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "P50"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "green",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 4,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 2,
      "options": {
        "legend": {
          "calcs": [
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "right",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PEB814BE3CA3B78C0"
          },
          "editorMode": "code",
          "expr": "histogram_quantile(n    0.99, sum(n        rate(n            nginx_http_request_duration_seconds_bucketn            {n                path=~"$path",n                status_code=~"$status",n                method=~"$method"n            }[1m])) by (le)n    )",
          "legendFormat": "P99",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PEB814BE3CA3B78C0"
          },
          "editorMode": "code",
          "expr": "histogram_quantile(n    0.90, sum(n        rate(n            nginx_http_request_duration_seconds_bucketn            {n                path=~"$path",n                status_code=~"$status",n                method=~"$method"n            }[1m])) by (le)n    )",
          "hide": false,
          "legendFormat": "P90",
          "range": true,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PEB814BE3CA3B78C0"
          },
          "editorMode": "code",
          "expr": "histogram_quantile(n    0.50, sum(n        rate(n            nginx_http_request_duration_seconds_bucketn            {n                path=~"$path",n                status_code=~"$status",n                method=~"$method"n            }[1m])) by (le)n    )",
          "hide": false,
          "legendFormat": "P50",
          "range": true,
          "refId": "C"
        }
      ],
      "title": "Latency",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "ea60a2ad-a704-42cf-8bcd-9e2e2dd02366"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "fixedColor": "orange",
            "mode": "fixed"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 50,
            "gradientMode": "opacity",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "smooth",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": true,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "reqps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 24,
        "x": 0,
        "y": 4
      },
      "id": 4,
      "options": {
        "legend": {
          "calcs": [
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "right",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PEB814BE3CA3B78C0"
          },
          "editorMode": "code",
          "expr": "n    sum(n        rate(n            nginx_request_status_code_total{n               }[5m]n            )n        ) by (path)",
          "legendFormat": "{{ path }}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Traffic",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "ea60a2ad-a704-42cf-8bcd-9e2e2dd02366"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "fixedColor": "green",
            "mode": "fixed"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 50,
            "gradientMode": "opacity",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "smooth",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": true,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "dashed"
            }
          },
          "decimals": 1,
          "mappings": [],
          "max": 1,
          "thresholds": {
            "mode": "percentage",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "dark-green",
                "value": 80
              }
            ]
          },
          "unit": "percentunit"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 24,
        "x": 0,
        "y": 8
      },
      "id": 6,
      "options": {
        "legend": {
          "calcs": [
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "right",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PEB814BE3CA3B78C0"
          },
          "editorMode": "code",
          "expr": "sum(rate(nginx_request_status_code_total{path=~"$path", status_code!~"[4-5].*"}[1m])) / sum(rate(nginx_request_status_code_total{path=~"$path"}[1m]))",
          "legendFormat": "{{ ingress }}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Availability (non-4|5xx responses)",
      "type": "timeseries"
    }
  ],
  "refresh": "5s",
  "schemaVersion": 38,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": true,
          "text": "/",
          "value": "/"
        },
        "definition": "label_values(nginx_http_request_duration_seconds_bucket,path)",
        "hide": 0,
        "includeAll": true,
        "multi": false,
        "name": "path",
        "options": [],
        "query": {
          "query": "label_values(nginx_http_request_duration_seconds_bucket,path)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": ".*",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      },
      {
        "current": {
          "selected": true,
          "text": "success",
          "value": "[2-3].*"
        },
        "hide": 0,
        "includeAll": false,
        "multi": false,
        "name": "status",
        "options": [
          {
            "selected": true,
            "text": "success",
            "value": "[2-3].*"
          },
          {
            "selected": false,
            "text": "error",
            "value": "[4-5].*"
          }
        ],
        "query": "success : [2-3].*,error : [4-5].*",
        "queryValue": "",
        "skipUrlSync": false,
        "type": "custom"
      },
      {
        "current": {
          "selected": true,
          "text": "All",
          "value": "$__all"
        },
        "hide": 0,
        "includeAll": true,
        "multi": false,
        "name": "method",
        "options": [
          {
            "selected": true,
            "text": "All",
            "value": "$__all"
          },
          {
            "selected": false,
            "text": "GET",
            "value": "GET"
          },
          {
            "selected": false,
            "text": "POST",
            "value": "POST"
          },
          {
            "selected": false,
            "text": "PUT",
            "value": "PUT"
          },
          {
            "selected": false,
            "text": "DELETE",
            "value": "DELETE"
          }
        ],
        "query": "GET,POST,PUT,DELETE",
        "queryValue": "",
        "skipUrlSync": false,
        "type": "custom"
      }
    ]
  },
  "time": {
    "from": "now-5m",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "Latency & Traffic & Errors",
  "uid": "0fk-M6O4k",
  "version": 3,
  "weekStart": ""
}

使用ab进行压测

进行压测图形如下