本文以3节点集群（1-1-1）,进行节点故障现象模拟。租户副本数量为2，模拟单节点故障，双节点故障现象。
三节点搭建文章地址：
https://www.modb.pro/db/190765
https://www.modb.pro/db/190799
1 系统信息
sys租户副本信息：为2分布在zone1,zone.
sys租户副本数量为2：
MySQL [oceanbase]> select * from __all_tenant where tenant_id=1\G;
*************************** 1. row ***************************gmt_create: 2021-12-07 16:42:12.523525gmt_modified: 2021-12-07 16:42:12.523525tenant_id: 1tenant_name: sysreplica_num: -1zone_list: zone1;zone2primary_zone: zone1;zone2locked: 0collation_type: 0info: system tenantread_only: 0rewrite_merge_version: 0locality: FULL{1}@zone1, FULL{1}@zone2logonly_replica_num: 0previous_locality:storage_format_version: 0
storage_format_work_version: 0default_tablegroup_id: -1compatibility_mode: 0drop_tenant_time: -1status: TENANT_STATUS_NORMALin_recyclebin: 0
1 row in set (0.001 sec)
ERROR: No query specified
2 observer信息
集群信息（1-1-1）
zone1 192.168.5.200 2881 2882
zone1 192.168.5.200 3881 3882
zone1 192.168.5.200 4881 4882
MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-07 15:52:48.009135 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-07 16:17:55.557006 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863788395674 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-07 16:24:29.110670 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ 3 rows in set (0.002 sec)
3 故障模拟：2个节点kill
对外不能提供服务，只有zone1,zone2正常才能正常提供服务。
3.1 kill zone2,zone2
[admin@db01 ~]$ ps -ef|grep observer |grep -v grep
admin 727 1 99 02:52 ? 20:43:00 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone1 -p 2881 -P 2882 -c 1 -d /data/observer01/store -i ens192 -l INFO
admin 1559 1 99 02:55 ? 19:04:39 /home/admin/oceanbase/bin/observer -r 192.168.5.200:3882:3881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO
admin 2141 1 80 02:55 ? 13:37:51 /home/admin/oceanbase/bin/observer -r 192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone3 -p 4881 -P 4882 -c 1 -d /data/observer03/store -i ens192 -l INFO
[admin@db01 ~]$ kill -9 1559 2141
[admin@db01 ~]$ ps -ef|grep observer |grep -v grep
admin 727 1 99 02:52 ? 20:43:22 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone1 -p 2881 -P 2882 -c 1 -d /data/observer01/store -i ens192 -l INFO
[admin@db01 ~]$
3.2 登陆数据库，发现系统已经不能正常提供服务
对外不能提供服务，只有zone1,zone2正常才能正常提供服务。
3.3 启动zone2
[admin@db01 ~]$ cd /data/observer02/ && /home/admin/oceanbase/bin/observer -r "192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881" -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO rs list: 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 optstr: __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 zone: zone2 mysql port: 3881 rpc port: 3882 cluster id: 1 data_dir: /data/observer02/store devname: ens192 log level: INFO [admin@db01 observer02]$ [admin@db01 observer02]$ ps -ef|grep obs|grep -v grep admin 727 1 99 02:52 ? 20:47:35 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone1 -p 2881 -P 2882 -c 1 -d /data/observer01/store -i ens192 -l INFO admin 11948 1 99 19:56 ? 00:00:16 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO
3.4 查询系统表，发现正常提供服务
[admin@db01 ~]$ sh c.sh Enter password: Welcome to the OceanBase. Commands end with ; or \g. Your MySQL connection id is 3221567989 Server version: 5.7.25 OceanBase 3.1.1 (r4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e) (Built Oct 21 2021 10:33:14) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [oceanbase]> select * from __all_server; ERROR 4012 (HY000): Timeout MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | blo ck_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | wit h_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-07 15:52:48.009135 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 08:52:52.289833 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 0 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-07 16:24:29.110670 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ 3 rows in set (3.090 sec) MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | blo ck_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | wit h_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-08 08:57:05.281970 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 08:57:11.281488 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638925030293827 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-08 08:57:05.278698 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ 3 rows in set (0.001 sec) MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-08 08:57:05.281970 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 08:57:11.281488 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638925030293827 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-08 08:57:05.278698 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ 3 rows in set (0.000 sec) MySQL [oceanbase]>[admin@db01 ~]$ sh c.sh Enter password: Welcome to the OceanBase. Commands end with ; or \g. Your MySQL connection id is 3221567989 Server version: 5.7.25 OceanBase 3.1.1 (r4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e) (Built Oct 21 2021 10:33:14) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [oceanbase]> select * from __all_server; ERROR 4012 (HY000): Timeout MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | blo ck_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | wit h_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-07 15:52:48.009135 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 08:52:52.289833 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 0 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-07 16:24:29.110670 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ 3 rows in set (3.090 sec) MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | blo ck_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | wit h_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-08 08:57:05.281970 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 08:57:11.281488 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638925030293827 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-08 08:57:05.278698 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ 3 rows in set (0.001 sec) MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-08 08:57:05.281970 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 08:57:11.281488 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638925030293827 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-08 08:57:05.278698 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ 3 rows in set (0.000 sec) MySQL [oceanbase]>
状态截图：
4 故障模拟：kill zone2
对外不能提供服务，只有zone1,zone2正常才能正常提供服务
4.1检查租户副本信息
sys租户副本分布在：locality: FULL{1}@zone1, FULL{1}@zone2
MySQL [oceanbase]> select * from __all_tenant where tenant_id=1\G;
*************************** 1. row ***************************gmt_create: 2021-12-07 16:42:12.523525gmt_modified: 2021-12-07 16:42:12.523525tenant_id: 1tenant_name: sysreplica_num: -1zone_list: zone1;zone2primary_zone: zone1;zone2locked: 0collation_type: 0info: system tenantread_only: 0rewrite_merge_version: 0locality: FULL{1}@zone1, FULL{1}@zone2logonly_replica_num: 0previous_locality:
storage_format_version: 0
storage_format_work_version: 0default_tablegroup_id: -1compatibility_mode: 0drop_tenant_time: -1status: TENANT_STATUS_NORMALin_recyclebin: 0
1 row in set (0.001 sec)
ERROR: No query specified
4.2模拟zone2故障
对外不能提供服务，只有zone1,zone2正常才能正常提供服务
sys租户副本分布在：locality: FULL{1}@zone1, FULL{1}@zone2
4.3故障现在，不能正常对外提供服务
4.4启动zone2
[admin@db01 observer03]$ [admin@db01 observer03]$ cd /data/observer02/ && /home/admin/oceanbase/bin/observer -r "192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881" -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO rs list: 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 optstr: __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 zone: zone2 mysql port: 3881 rpc port: 3882 cluster id: 1 data_dir: /data/observer02/store devname: ens192 log level: INFO [admin@db01 observer02]$ ps -ef|grep 3882 admin 15281 1 99 20:25 ? 00:07:57 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO admin 16343 14059 0 20:32 pts/3 00:00:00 grep --color=auto 3882
4.5查看服务状态
zone2 的observer2启动之后，可以正常查询。
MySQL [oceanbase]> select * from __all_server; ERROR 4012 (HY000): Timeout MySQL [oceanbase]> select * from __all_server; ERROR 4012 (25000): Statement is timeout MySQL [oceanbase]> select * from __all_tenant where tenant_id=1\G; ERROR 4012 (25000): Statement is timeout ERROR: No query specified MySQL [oceanbase]> select * from __all_server; ERROR 4012 (HY000): Timeout MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-08 09:26:07.204402 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 09:26:07.207814 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 0 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-08 09:26:07.208028 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638926371893289 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ 3 rows in set (0.001 sec)
5 故障模拟：kill zone3
对外正常提供服务。
5.1检查server状态
MySQL [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| 2021-12-07 15:52:36.583969 | 2021-12-08 09:26:07.204402 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 |
| 2021-12-07 15:56:07.681043 | 2021-12-08 09:26:15.207936 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638926774220189 | 0 | 1 | 0 |
| 2021-12-07 15:56:11.908905 | 2021-12-08 09:26:07.208028 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638926371893289 | 0 | 1 | 0 |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
3 rows in set (0.001 sec)
5.2 模拟zone3 故障
[admin@db01 observer02]$
[admin@db01 observer02]$ ps -ef|grep '\-P 4882'|grep -v grep
admin 14080 1 90 20:17 ? 00:19:29 /home/admin/oceanbase/bin/observer -r 192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone3 -p 4881 -P 4882 -c 1 -d /data/observer03/store -i ens192 -l INFO
[admin@db01 observer02]$ kill -9 14080
[admin@db01 observer02]$ ps -ef|grep '\-P 4882'|grep -v grep
5.3 查询系统表
数据库状态正常，对外正常提供服务
MySQL [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| 2021-12-07 15:52:36.583969 | 2021-12-08 09:26:07.204402 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 |
| 2021-12-07 15:56:07.681043 | 2021-12-08 09:26:15.207936 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638926774220189 | 0 | 1 | 0 |
| 2021-12-07 15:56:11.908905 | 2021-12-08 09:39:00.647693 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 0 | 0 | 1 | 0 |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
3 rows in set (0.001 sec)
对外正常提供服务。
数据库状态正常，对外正常提供服务
