OBAgent 简介
OBAgent 是用 GO 语言开发的监控采集框架，通常部署在 OBServer 节点上。OBAgent 支持推、拉两种数据采集模式，可以满足不同的应用场景。OBAgent 默认支持的插件包括主机数据采集、OceanBase 数据库指标的采集、监控数据标签处理和 Prometheus 协议的 HTTP 服务。要使 OBAgent 支持其他数据源的采集，或者自定义数据的处理流程，您只需要开发对应的插件即可。
编辑 OBAgent 部署配置文件
OBAgent 部署配置文件可以跟 OceanBase 集群部署配置文件一起，也可以后期单独部署。附录 A.1 展示了同时部署 OceanBase 集群和 OBAgent。
下面示例是采用单独的配置文件部署 OBAgent 。OBAgent 的部署配置文件风格跟 OceanBase 集群部署配置文件一样。 首先是指定部署节点，包括节点名称和 IP 。节点名称保持唯一就行，可以是主机名（假设主机名是唯一的）。 然后指定全局配置。各个节点共同的配置都放在 global 节下。节点定制化的配置就不用放在这个下面。 然后指定各个节点定制化的配置。比如说每个节点的 zone 名称是不一样的。其他的根据实际情况填写。
vim obagent-only.yaml obagent: servers: - name: obce01 # Please don't use hostname, only IP can be supported ip: 172.20.249.53 - name: obce02 ip: 172.20.249.55 - name: obce03 ip: 172.20.249.56 global: # The working directory for obagent. obagent is started under this directory. This is a required field. home_path: /home/admin/obagent # The port that pulls and manages the metrics. The default port number is 8088. server_port: 8088 # Debug port for pprof. The default port number is 8089. pprof_port: 8089 sql_port: 2881 rpc_port: 2882 # Log level. The default value is INFO. log_level: INFO # Log path. The default value is log/monagent.log. log_path: log/monagent.log # Encryption method. OBD supports aes and plain. The default value is plain. crypto_method: plain # Path to store the crypto key. The default value is conf/.config_secret.key. # crypto_path: conf/.config_secret.key # Size for a single log file. Log size is measured in Megabytes. The default value is 30M. log_size: 30 # Expiration time for logs. The default value is 7 days. log_expire_day: 7 # The maximum number for log files. The default value is 10. log_file_count: 10 # Whether to use local time for log files. The default value is true. # log_use_localtime: true # Whether to enable log compression. The default value is true. # log_compress: true # Username for HTTP authentication. The default value is admin. http_basic_auth_user: admin # Password for HTTP authentication. The default value is root. http_basic_auth_password: eIYf7NAZeT # Username for debug service. The default value is admin. pprof_basic_auth_user: admin # Password for debug service. The default value is root. pprof_basic_auth_password: eIYf7NAZeT # 以下配置必须与 OceanBase 数据库一致 # Monitor username for OceanBase Database. The user must have read access to OceanBase Database as a system tenant. The default value is root. monitor_user: monitor # Monitor password for OceanBase Database. The default value is empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the root_password in oceanbase-ce. monitor_password: fLyaqjrp2R # Cluster name for OceanBase Database. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the appname in oceanbase-ce. cluster_name: obce-3zones # Cluster ID for OceanBase Database. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the cluster_id in oceanbase-ce. cluster_id: 1 obce01: zone: zone1 obce02: zone: zone2 obce03: zone: zone3
注意：
- 指定节点的连接端口用的是
sql_port不是mysql_port，这点跟 OBSERVER 节点配置不一样。 - 监控用户（
monitor_user对应）和密码需要在 SYS 租户下创建。grant select on oceanbase.* to monitor identified by 'fLyaqjrp2R';。
OBD 部署 OBAgent
第一次使用 deploy 命令，指定 OBAgent 的配置文件。
[admin@obce00 ~]$ obd cluster deploy obagent-only -c obagent-only.yaml obagent-1.0.0 already installed. +---------------------------------------------------------------------------+ | Packages | +------------+---------+---------+------------------------------------------+ | Repository | Version | Release | Md5 | +------------+---------+---------+------------------------------------------+ | obagent | 1.0.0 | 2.el8 | 1d65fc3d2cd08b26d6142b6149eb6806260aa7db | +------------+---------+---------+------------------------------------------+ Repository integrity check ok Parameter check ok Open ssh connection ok Remote obagent-1.0.0-1d65fc3d2cd08b26d6142b6149eb6806260aa7db repository install ok Remote obagent-1.0.0-1d65fc3d2cd08b26d6142b6149eb6806260aa7db repository lib check ok Cluster status check ok Initializes obagent work home ok obagent-only deployed [admin@obce00 ~]$ [admin@obce00 ~]$ obd cluster list +------------------------------------------------------------------------+ | Cluster List | +--------------+---------------------------------------+-----------------+ | Name | Configuration Path | Status (Cached) | +--------------+---------------------------------------+-----------------+ | obce-3zones | /home/admin/.obd/cluster/obce-3zones | running | | obagent-only | /home/admin/.obd/cluster/obagent-only | deployed | +--------------+---------------------------------------+-----------------+
上面 deploy 命令运行后，配置文件就被复制到 ~/.obd/cluster/obagent-only/config.yaml 了。后续修改 obagent-only.yaml 文件就不会生效。此时可以采取 edit-config 编辑使用的配置文件，或者使用 destroy 命令清理部署，重新读取 obagent-only.yaml 开始部署。这个取决于改动的影响范围。
deploy 命令只是在各个节点上部署 OBAgent 软件（直接解压缩方式，不是 RPM 安装），目录如下：
[admin@obce01 ~]$ pwd /home/admin [admin@obce01 ~]$ tree obagent/ obagent/ ├── bin │ └── monagent -> /home/admin/.obd/repository/obagent/1.0.0/1d65fc3d2cd08b26d6142b6149eb6806260aa7db/bin/monagent ├── conf │ ├── config_properties │ │ ├── monagent_basic_auth.yaml │ │ └── monagent_pipeline.yaml │ ├── module_config │ │ ├── monagent_basic_auth.yaml │ │ ├── monagent_config.yaml │ │ ├── monitor_node_host.yaml │ │ └── monitor_ob.yaml │ ├── monagent.yaml │ └── prometheus_config │ ├── prometheus.yaml │ └── rules │ ├── host_rules.yaml │ └── ob_rules.yaml ├── lib ├── log │ └── monagent.log └── run 9 directories, 12 files [admin@obce01 ~]$
OBD 启动 OBAgent
启动命令是 start 。
[admin@obce00 ~]$ obd cluster start obagent-only Get local repositories and plugins ok Open ssh connection ok Cluster param config check ok Check before start obagent ok obagent program health check ok +---------------------------------------------------+ | obagent | +---------------+-------------+------------+--------+ | ip | server_port | pprof_port | status | +---------------+-------------+------------+--------+ | 172.20.249.53 | 8088 | 8089 | active | | 172.20.249.55 | 8088 | 8089 | active | | 172.20.249.56 | 8088 | 8089 | active | +---------------+-------------+------------+--------+ obagent-only running [admin@obce00 ~]$ [admin@obce00 ~]$ obd cluster list +------------------------------------------------------------------------+ | Cluster List | +--------------+---------------------------------------+-----------------+ | Name | Configuration Path | Status (Cached) | +--------------+---------------------------------------+-----------------+ | obce-3zones | /home/admin/.obd/cluster/obce-3zones | running | | obagent-only | /home/admin/.obd/cluster/obagent-only | running | +--------------+---------------------------------------+-----------------+ [admin@obce00 ~]$
OBAgent 启动后有两个进程，其中进程 moagent 会监听指定端口。
[admin@obce01 ~]$ ps -ef|grep agent | grep -v grep admin 90855 1 0 12:08 ? 00:00:00 /home/admin/obagent/bin/monagent -c conf/monagent.yaml [admin@obce01 ~]$ [admin@obce01 ~]$ netstat -ntlp |grep 90855 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp6 0 0 :::8088 :::* LISTEN 90855/monagent tcp6 0 0 :::8089 :::* LISTEN 90855/monagent [admin@obce01 ~]$
事后也可以通过 OBD 查看 OBAgent 部署情况。
[admin@obce00 ~]$ obd cluster display obagent-only Get local repositories and plugins ok Open ssh connection ok Cluster status check ok +---------------------------------------------------+ | obagent | +---------------+-------------+------------+--------+ | ip | server_port | pprof_port | status | +---------------+-------------+------------+--------+ | 172.20.249.53 | 8088 | 8089 | active | | 172.20.249.55 | 8088 | 8089 | active | | 172.20.249.56 | 8088 | 8089 | active | +---------------+-------------+------------+--------+ [admin@obce00 ~]$
Prometheus 配置
OBAgent 启动后会在节点自动生成 Prometheus 配置文件, 位置在 OBAgent 安装目录下，如 /home/admin/obagent/conf/prometheus_config/ 。这个配置文件可以给 Prometheus 产品直接使用。
示例如下：
vim prometheus_config/prometheus.yaml global: scrape_interval: 1s evaluation_interval: 10s rule_files: - "rules/*rules.yaml" scrape_configs: - job_name: prometheus metrics_path: /metrics scheme: http static_configs: - targets: - 'localhost:9090' - job_name: node basic_auth: username: admin password: eIYf7NAZeT metrics_path: /metrics/node/host scheme: http static_configs: - targets: - 172.20.249.53:8088 - 172.20.249.55:8088 - 172.20.249.56:8088 - job_name: ob_basic basic_auth: username: admin password: eIYf7NAZeT metrics_path: /metrics/ob/basic scheme: http static_configs: - targets: - 172.20.249.53:8088 - 172.20.249.55:8088 - 172.20.249.56:8088 - job_name: ob_extra basic_auth: username: admin password: eIYf7NAZeT metrics_path: /metrics/ob/extra scheme: http static_configs: - targets: - 172.20.249.53:8088 - 172.20.249.55:8088 - 172.20.249.56:8088 - job_name: agent basic_auth: username: admin password: eIYf7NAZeT metrics_path: /metrics/stat scheme: http static_configs: - targets: - 172.20.249.53:8088 - 172.20.249.55:8088 - 172.20.249.56:8088
稍加说明如下：
下载解压缩 prometheus 后，启动方法
cd prometheus-2.30.3.linux-amd64 && ./prometheus ./prometheus.yaml level=info ts=2021-11-19T05:41:57.789Z caller=main.go:400 msg="No time or size retention was set so using the default time retention" duration=15d level=info ts=2021-11-19T05:41:57.789Z caller=main.go:438 msg="Starting Prometheus" version="(version=2.30.3, branch=HEAD, revision=f29caccc42557f6a8ec30ea9b3c8c089391bd5df)" level=info ts=2021-11-19T05:41:57.789Z caller=main.go:443 build_context="(go=go1.17.1, user=root@5cff4265f0e3, date=20211005-16:10:52)" level=info ts=2021-11-19T05:41:57.789Z caller=main.go:444 host_details="(Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 obce00 (none))" level=info ts=2021-11-19T05:41:57.789Z caller=main.go:445 fd_limits="(soft=65535, hard=65535)" level=info ts=2021-11-19T05:41:57.789Z caller=main.go:446 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2021-11-19T05:41:57.791Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2021-11-19T05:41:57.792Z caller=main.go:822 msg="Starting TSDB ..." level=info ts=2021-11-19T05:41:57.792Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false level=info ts=2021-11-19T05:41:57.795Z caller=head.go:479 component=tsdb msg="Replaying on-disk memory mappable chunks if any" level=info ts=2021-11-19T05:41:57.795Z caller=head.go:513 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=7.579µs level=info ts=2021-11-19T05:41:57.795Z caller=head.go:519 component=tsdb msg="Replaying WAL, this may take a while" level=info ts=2021-11-19T05:41:57.795Z caller=head.go:590 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 level=info ts=2021-11-19T05:41:57.795Z caller=head.go:596 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.235µs wal_replay_duration=183.559µs total_replay_duration=238.771µs level=info ts=2021-11-19T05:41:57.796Z caller=main.go:849 fs_type=EXT4_SUPER_MAGIC level=info ts=2021-11-19T05:41:57.796Z caller=main.go:852 msg="TSDB started" level=info ts=2021-11-19T05:41:57.796Z caller=main.go:979 msg="Loading configuration file" filename=prometheus.yaml level=info ts=2021-11-19T05:41:57.802Z caller=main.go:1016 msg="Completed loading of configuration file" filename=prometheus.yaml totalDuration=6.363415ms db_storage=1.001µs remote_storage=4.404µs web_handler=3.04µs query_engine=886ns scrape=418.015µs scrape_sd=129.011µs notify=5.433µs notify_sd=4.872µs rules=5.284804ms level=info ts=2021-11-19T05:41:57.802Z caller=main.go:794 msg="Server is ready to receive web requests."
启动后通过浏览器访问：http://172.24.50.39:9090/graph 。
具体 Prometheus 使用方法可以参考 Prometheus 官方问答 。
OBAgent 重启方法
直接重启某个节点的 OBAgent 方法是：
kill -9 `pidof monagent` cd /home/admin/obagent && nohup bin/monagent -c conf/monagent.yaml &
如果是集中重启，那就使用 OBD 命令：
obd cluster restart obagent_only
如果 OBAgent 是跟 OceanBase 一起部署的，那只能重启组件 obagent 。
obd cluster restart obce-3zones-obagent -c obagent [admin@obce00 ~]$ obd cluster restart obce-3zones-obagent -c obagent Get local repositories and plugins ok Open ssh connection ok Stop obagent ok succeed Get local repositories and plugins ok Open ssh connection ok Cluster param config check ok Check before start obagent ok obagent program health check ok +--------------------------------------------------+ | obagent | +--------------+-------------+------------+--------+ | ip | server_port | pprof_port | status | +--------------+-------------+------------+--------+ | 172.24.50.37 | 8088 | 8089 | active | | 172.24.50.40 | 8088 | 8089 | active | | 172.24.50.38 | 8088 | 8089 | active | +--------------+-------------+------------+--------+ succeed
Grafana 使用
首先请从 Grafana 官网下载最新版本，并安装启动。下载地址：https://grafana.com/grafana/download?pg=get&plcmt=selfmanaged-box1-cta1 。
然后在 Grafana 里新增 Datasource，填入 Prometheus 地址。
第三，从 Grafana 官网下载 OceanBase 提交的 主机性能模板和 OceanBase 性能模板文件，文件是 json 格式。
- 主机性能模板
- OceanBase 性能模板
下载到本机后，在Grafana 里 Import 这两个 json 文件。
附录
A.1 OceanBase 和 OBAgent 一起的配置文件
# Only need to configure when remote login is required
user:
username: admin
# password: your password if need
key_file: /home/admin/.ssh/id_rsa.pub
port: your ssh port, default 22
# timeout: ssh connection timeout (second), default 30
oceanbase-ce:
servers:
- name: obce01
# Please don't use hostname, only IP can be supported
ip: 172.24.50.37
- name: obce02
ip: 172.24.50.40
- name: obce03
ip: 172.24.50.38
global:
# Please set devname as the network adaptor's name whose ip is in the setting of severs.
# if set severs as "127.0.0.1", please set devname as "lo"
# if current ip is 192.168.1.10, and the ip's network adaptor's name is "eth0", please use "eth0"
devname: eth0
cluster_id: 2
# please set memory limit to a suitable value which is matching resource.
memory_limit: 8G # The maximum running memory for an observer
system_memory: 3G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
stack_size: 512K
cpu_count: 16
cache_wash_threshold: 1G
__min_full_resource_pool_memory: 268435456
workers_per_cpu_quota: 10
schema_history_expire_time: 1d
# The value of net_thread_count had better be same as cpu's core number.
net_thread_count: 4
major_freeze_duty_time: Disable
minor_freeze_times: 10
enable_separate_sys_clog: 0
enable_merge_by_turn: FALSE
#datafile_disk_percentage: 20 # The percentage of the data_dir space to the total disk space. This value takes effect only when datafile_size is 0. The default value is 90.
datafile_size: 50G
syslog_level: WARN # System log level. The default value is INFO.
enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.
max_syslog_file_count: 10 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.
# observer cluster name, consistent with obproxy's cluster_name
appname: obce-3zones
root_password: 0EI5N08d # root user password, can be empty
proxyro_password: uY7Yf8zx # proxyro user pasword, consistent with obproxy's observer_sys_password, can be empty
obce01:
mysql_port: 3881 # External port for OceanBase Database. The default value is 3881.
rpc_port: 3882 # Internal port for OceanBase Database. The default value is 3882.
# The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
home_path: /home/admin/oceanbase-ce
# The directory for data storage. The default value is $home_path/store.
data_dir: /data/obce
# The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
redo_dir: /redo/obce
zone: zone1
obce02:
mysql_port: 3881 # External port for OceanBase Database. The default value is 3881.
rpc_port: 3882 # Internal port for OceanBase Database. The default value is 3882.
# The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
home_path: /home/admin/oceanbase-ce
# The directory for data storage. The default value is $home_path/store.
data_dir: /data/obce
# The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
redo_dir: /redo/obce
zone: zone2
obce03:
mysql_port: 3881 # External port for OceanBase Database. The default value is 3881.
rpc_port: 3882 # Internal port for OceanBase Database. The default value is 3882.
# The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
home_path: /home/admin/oceanbase-ce
# The directory for data storage. The default value is $home_path/store.
data_dir: /data/obce
# The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
redo_dir: /redo/obce
zone: zone3
obproxy:
servers:
- 172.24.50.39
- 172.24.50.37
# Set ependent components for the component.
# When the associated configurations are not done, OBD will automatically get the these configurations from the dependent components.
depends:
- oceanbase-ce
global:
listen_port: 3883 # External port. The default value is 3883.
prometheus_listen_port: 3884 # The Prometheus port. The default value is 3884.
home_path: /home/admin/obproxy
# oceanbase root server list
# format: ip:mysql_port;ip:mysql_port
rs_list: 172.24.50.37:3881;172.24.50.40:3881;172.24.50.38:3881
enable_cluster_checkout: false
# observer cluster name, consistent with oceanbase-ce's appname
cluster_name: obce-3zones
obproxy_sys_password: 0MdTv1tm # obproxy sys user password, can be empty
observer_sys_password: uY7Yf8zx # proxyro user pasword, consistent with oceanbase-ce's proxyro_password, can be empty
obagent:
servers:
- name: obce01
# Please don't use hostname, only IP can be supported
ip: 172.24.50.37
- name: obce02
ip: 172.24.50.40
- name: obce03
ip: 172.24.50.38
depends:
- oceanbase-ce
global:
# The working directory for obagent. obagent is started under this directory. This is a required field.
home_path: /home/admin/obagent
# The port that pulls and manages the metrics. The default port number is 8088.
server_port: 8088
# Debug port for pprof. The default port number is 8089.
pprof_port: 8089
sql_port: 3881
rpc_port: 3882
# Log level. The default value is INFO.
log_level: INFO
# Log path. The default value is log/monagent.log.
log_path: log/monagent.log
# Encryption method. OBD supports aes and plain. The default value is plain.
crypto_method: plain
# Path to store the crypto key. The default value is conf/.config_secret.key.
# crypto_path: conf/.config_secret.key
# Size for a single log file. Log size is measured in Megabytes. The default value is 30M.
log_size: 30
# Expiration time for logs. The default value is 7 days.
log_expire_day: 7
# The maximum number for log files. The default value is 10.
log_file_count: 10
# Whether to use local time for log files. The default value is true.
# log_use_localtime: true
# Whether to enable log compression. The default value is true.
# log_compress: true
# Username for HTTP authentication. The default value is admin.
http_basic_auth_user: admin
# Password for HTTP authentication. The default value is root.
http_basic_auth_password: eIYf7NAZeT
# Username for debug service. The default value is admin.
pprof_basic_auth_user: admin
# Password for debug service. The default value is root.
pprof_basic_auth_password: eIYf7NAZeT
# 以下配置必须与 OceanBase 数据库一致
# Monitor username for OceanBase Database. The user must have read access to OceanBase Database as a system tenant. The default value is root.
monitor_user: monitor
# Monitor password for OceanBase Database. The default value is empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the root_password in oceanbase-ce.
monitor_password: fLyaqjrp2R
# Cluster name for OceanBase Database. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the appname in oceanbase-ce.
# cluster_name: obce-3zones
# Cluster ID for OceanBase Database. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the cluster_id in oceanbase-ce.
# cluster_id: 2
A.2 obagent 输出的性能数据
[admin@obce00 ~]$ curl --user admin:eIYf7NAZeT -L 'http://172.24.50.40:8088/metrics/ob/basic'
# HELP ob_active_session_num monitor collected metric
# TYPE ob_active_session_num untyped
ob_active_session_num{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 6
# HELP ob_cache_size_bytes monitor collected metric
# TYPE ob_cache_size_bytes untyped
ob_cache_size_bytes{app="OB",cache_name="location_cache",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 4.193216e+06
ob_cache_size_bytes{app="OB",cache_name="user_tab_col_stat_cache",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 6.290304e+06
ob_cache_size_bytes{app="OB",cache_name="user_table_stat_cache",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 6.290304e+06
# HELP ob_partition_num monitor collected metric
# TYPE ob_partition_num untyped
ob_partition_num{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",role="1",svr_ip="172.24.50.40",tenant_name="sys"} 1198
# HELP ob_plan_cache_access_total monitor collected metric
# TYPE ob_plan_cache_access_total untyped
ob_plan_cache_access_total{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 22984
# HELP ob_plan_cache_hit_total monitor collected metric
# TYPE ob_plan_cache_hit_total untyped
ob_plan_cache_hit_total{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 8645
# HELP ob_plan_cache_memory_bytes monitor collected metric
# TYPE ob_plan_cache_memory_bytes untyped
ob_plan_cache_memory_bytes{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 1.3404239e+07
# HELP ob_server_num monitor collected metric
# TYPE ob_server_num untyped
ob_server_num{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",server_ips="172.24.50.37,172.24.50.38,172.24.50.40",status="active",svr_ip="172.24.50.40"} 3
# HELP ob_sysstat monitor collected metric
# TYPE ob_sysstat untyped
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="10000",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 47136
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="10001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 3.0078186e+07
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="10002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 46863
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="10003",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 4.291008e+07
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="10005",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} -2.050408e+06
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="10006",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 4096
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="130000",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 8.59442e+07
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="130001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 8.8080384e+07
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="130002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 4.0265315e+08
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="130004",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 8.0530635e+08
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="140002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 1.610612736e+09
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="140003",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 3.81681664e+08
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="140005",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 500
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="140006",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 1
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="20001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 4122
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="20002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 104938
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="30000",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="30001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="30002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="30005",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 15330
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="30006",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 7.566136e+06
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40000",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 463
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 1.863916e+06
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40003",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40004",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40005",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40006",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40007",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40008",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40009",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40010",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 108
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40011",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 8339
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40012",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 329
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="50000",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="50001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="50008",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="50009",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60000",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60003",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 4
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60004",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 43325
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60005",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 8.388608e+06
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60019",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60020",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60021",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60022",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60023",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60024",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="80040",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 11980
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="80041",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 2.0937394e+07
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="80057",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
# HELP ob_table_num monitor collected metric
# TYPE ob_table_num untyped
ob_table_num{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 1013
# HELP ob_waitevent_wait_seconds_total monitor collected metric
# TYPE ob_waitevent_wait_seconds_total untyped
ob_waitevent_wait_seconds_total{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 49578.6026
# HELP ob_waitevent_wait_total monitor collected metric
# TYPE ob_waitevent_wait_total untyped
ob_waitevent_wait_total{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 787473
# HELP ob_zone_current_timestamp monitor collected metric
# TYPE ob_zone_current_timestamp untyped
ob_zone_current_timestamp{app="OB",name="all_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="all_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="all_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="cluster",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="config_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="frozen_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="frozen_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="gc_schema_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="global_broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="idc",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="idc",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="idc",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merge_error",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merge_timeout",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merge_timeout",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merge_timeout",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="lease_info_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="privilege_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="proposal_frozen_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="recovery_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="recovery_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="recovery_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="region",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="region",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="region",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="snapshot_gc_ts",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="storage_format_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="storage_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="storage_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="storage_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="suspend_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="suspend_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="suspend_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="time_zone_info_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="try_frozen_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="warm_up_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="zone_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="zone_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="zone_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
# HELP ob_zone_stat monitor collected metric
# TYPE ob_zone_stat untyped
ob_zone_stat{app="OB",name="all_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1
ob_zone_stat{app="OB",name="all_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1
ob_zone_stat{app="OB",name="all_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1
ob_zone_stat{app="OB",name="broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1
ob_zone_stat{app="OB",name="broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1
ob_zone_stat{app="OB",name="broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1
ob_zone_stat{app="OB",name="cluster",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="config_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.63730315985165e+15
ob_zone_stat{app="OB",name="frozen_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="frozen_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1
ob_zone_stat{app="OB",name="gc_schema_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="global_broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1
ob_zone_stat{app="OB",name="idc",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="idc",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="idc",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="is_merge_error",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="is_merge_timeout",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="is_merge_timeout",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="is_merge_timeout",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="is_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="is_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="is_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="last_merged_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303150866399e+15
ob_zone_stat{app="OB",name="last_merged_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303150867716e+15
ob_zone_stat{app="OB",name="last_merged_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303150868863e+15
ob_zone_stat{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1
ob_zone_stat{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1
ob_zone_stat{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1
ob_zone_stat{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1
ob_zone_stat{app="OB",name="lease_info_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.63730315985492e+15
ob_zone_stat{app="OB",name="merge_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303150866399e+15
ob_zone_stat{app="OB",name="merge_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303150867716e+15
ob_zone_stat{app="OB",name="merge_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303150868863e+15
ob_zone_stat{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="privilege_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="proposal_frozen_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1
ob_zone_stat{app="OB",name="recovery_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="recovery_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="recovery_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="region",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="region",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="region",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="snapshot_gc_ts",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 2
ob_zone_stat{app="OB",name="status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 2
ob_zone_stat{app="OB",name="status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 2
ob_zone_stat{app="OB",name="storage_format_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 4
ob_zone_stat{app="OB",name="storage_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="storage_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="storage_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="suspend_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="suspend_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="suspend_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="time_zone_info_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="try_frozen_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1
ob_zone_stat{app="OB",name="warm_up_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="zone_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="zone_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="zone_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
附录：
- 第二章 2.1 OceanBase 部署准备
- 第二章 2.2 如何快速体验 OceanBase
- 第二章 2.3 如何规划部署 OceanBase 集群
- 第二章 2.4 如何初始化服务器环境
- 第二章 2.5 安装 OBD 自动化部署软件
- 第二章 2.6 使用 OBD 自动化部署单节点集群
- 第二章 2.7 使用OBD自动化部署多节点集群
- 第二章 2.8 参看和修改 OceanBase 参数集群
- 第二章 2.9 如何部署 OBAgent
- 第二章 2.10 如何重启 OceanBase 集群
- 第二章 2.11（高级）如何手动部署 OceanBase 集群
- 第二章 2.12 常见问题
- 第二章 2.13 附录
结束语
OceanBase社区版入门到实战教程直播正在进行中~
快和小伙伴一起进群交流学习吧~
加入直播群方式一：
钉钉群号3255 4020
加入直播群方式二：
扫码下方二维码加入
