作者：coredumped，资深数据库从业者，熟悉 MySQL 及分布式架构，为多家大型国有企业提供过技术支持、咨询、架构设计及培训工作。
引言
自OB 社区版开源以来，本人认为分布式数据库最吸引人的还是多节点部署和扩缩容操作，本文将从以下几方面对三节点部署和扩缩容操作做一个介绍。
- 环境信息
- 搭建三节点(1-1-1)
- 创建资源池和租户
- 查看数据分布
- 服务器扩容(1-1-1->2-2-2)
- 租户扩容
- 租户缩容
环境信息
搭建三节点(1-1-1)
使用 obd 快速部署。
配置文件：
cat distributed-with-obproxy.yaml ## Only need to configure when remote login is required user:username: rootpassword: Dhy123456 # key_file: your ssh-key file path if need # port: your ssh port, default 22 # timeout: ssh connection timeout (second), default 30 oceanbase-ce:servers:- name: z1# Please don't use hostname, only IP can be supportedip: 10.140.114.12- name: z2ip: 10.140.60.14- name: z3ip: 10.140.118.7global:# Please set devname as the network adaptor's name whose ip is in the setting of severs.# if set severs as "127.0.0.1", please set devname as "lo"# if current ip is 192.168.1.10, and the ip's network adaptor's name is "eth0", please use "eth0"devname: eth0# if current hardware's memory capacity is smaller than 50G, please use the setting of "mini-single-example.yaml" and do a small adjustment.memory_limit: 14Gsystem_memory: 4Gstack_size: 512Kcpu_count: 16cache_wash_threshold: 1G__min_full_resource_pool_memory: 268435456workers_per_cpu_quota: 10schema_history_expire_time: 1d# The value of net_thread_count had better be same as cpu's core number.net_thread_count: 4major_freeze_duty_time: Disableminor_freeze_times: 10enable_separate_sys_clog: 0enable_merge_by_turn: FALSEdatafile_disk_percentage: 20syslog_level: INFOenable_syslog_recycle: truemax_syslog_file_count: 4cluster_id: 1# observer cluster name, consistent with obproxy's cluster_nameappname: ob_cluster# root_password: # root user password, can be empty# proxyro_password: # proxyro user pasword, consistent with obproxy's observer_sys_password, can be empty# In this example , support multiple ob process in single node, so different process use different ports.# If deploy ob cluster in multiple nodes, the port and path setting can be same.z1:mysql_port: 2881rpc_port: 2882home_path: /mydata/observer1zone: zone1z2:mysql_port: 2881rpc_port: 2882home_path: /mydata/observer1zone: zone2z3:mysql_port: 2881rpc_port: 2882home_path: /mydata/observer1zone: zone3 obproxy:servers:- 10.140.114.12global:listen_port: 2883prometheus_listen_port: 2884home_path: /root/obproxy# oceanbase root server list# format: ip:mysql_port,ip:mysql_portrs_list: 10.140.114.12:2881;10.140.60.14:2881;10.140.118.7:2881#rs_list: 139.198.32.21:2881;139.198.32.253:2881;139.198.30.3:2881enable_cluster_checkout: false# observer cluster name, consistent with oceanbase-ce's appnamecluster_name: ob_cluster# obproxy_sys_password: # obproxy sys user password, can be empty# observer_sys_password: # proxyro user pasword, consistent with oceanbase-ce's proxyro_password, can be empty
obd 部署：
obd cluster deploy cluster -c distributed-with-obproxy.yaml Update OceanBase-community-stable-el8 ok Update OceanBase-development-kit-el8 ok oceanbase-ce-3.1.0 already installed. obproxy-3.1.0 already installed. +-----------------------------------------------------------------------------+ | Packages | +--------------+---------+---------+------------------------------------------+ | Repository | Version | Release | Md5 | +--------------+---------+---------+------------------------------------------+ | oceanbase-ce | 3.1.0 | 2.el8 | 3052acd41b981aed85ce858174e90f07d804f85b | | obproxy | 3.1.0 | 1.el8 | d242ea5fe45222b8f61c3135ba2aaa778c61ea22 | +--------------+---------+---------+------------------------------------------+ Open ssh connection ok Remote oceanbase-ce-3.1.0-3052acd41b981aed85ce858174e90f07d804f85b repository install ok Remote oceanbase-ce-3.1.0-3052acd41b981aed85ce858174e90f07d804f85b repository lib check ok Remote obproxy-3.1.0-d242ea5fe45222b8f61c3135ba2aaa778c61ea22 repository install ok Remote obproxy-3.1.0-d242ea5fe45222b8f61c3135ba2aaa778c61ea22 repository lib check ok Cluster status check ok z1(10.140.114.12) initializes cluster work home ^[[Bz2(10.140.60.14) initializes cluster work home z3(10.140.118.7) initializes cluster work home 10.140.114.12 init cluster work home cluster deployed
启动集群：
obd cluster start cluster Get local repositories and plugins ok Open ssh connection ok Cluster param config check ok Check before start observer ok Check before start obproxy ok Start observer ok observer program health check ok Connect to observer ok Initialize cluster Cluster bootstrap ok Wait for observer init ok +-------------------------------------------------+ | observer | +---------------+---------+------+-------+--------+ | ip | version | port | zone | status | +---------------+---------+------+-------+--------+ | 10.140.114.12 | 3.1.0 | 2881 | zone1 | active | | 10.140.118.7 | 3.1.0 | 2881 | zone3 | active | | 10.140.60.14 | 3.1.0 | 2881 | zone2 | active | +---------------+---------+------+-------+--------+ Start obproxy ok obproxy program health check ok Connect to obproxy ok Initialize cluster +-------------------------------------------------+ | obproxy | +---------------+------+-----------------+--------+ | ip | port | prometheus_port | status | +---------------+------+-----------------+--------+ | 10.140.114.12 | 2883 | 2884 | active | +---------------+------+-----------------+--------+ cluster running
创建资源池和租户
通过 proxy 连接到 observer， 创建资源及租户。
obclient -u root -p -h 127.0.0.1 -P 2883 Enter password: Welcome to the OceanBase. Commands end with ; or \g. Your MySQL connection id is 5 Server version: 5.6.25 OceanBase 3.1.0 (r-00672c3c730c3df6eef3b359eae548d8c2db5ea2) (Built Jun 29 2021 02:14:01) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> use oceanbase; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MySQL [oceanbase]> 创建资源池，unit数量为1 MySQL [oceanbase]> create resource unit S2 max_cpu=2, min_cpu=2, max_memory='4G', min_memory='2G', max_iops=10000, min_iops=1000, max_session_num=1000000, max_disk_size='50G'; Query OK, 0 rows affected (0.009 sec) MySQL [oceanbase]> create resource pool pool_2 unit='S2', unit_num=1; Query OK, 0 rows affected (0.021 sec) MySQL [oceanbase]> CREATE TENANT IF NOT EXISTS tenant_2 -> charset='utf8mb4', -> replica_num=3, -> zone_list=('zone1','zone2','zone3'), -> primary_zone='RANDOM', -> resource_pool_list=('pool_2'); Query OK, 0 rows affected (1.477 sec)
通过自己创建的租户 tenant_2 登录创建表。
[root@dhy01 ~]# obclient -u root@tenant_2 -p -h 127.0.0.1 -P 2881 Enter password: Welcome to the OceanBase. Commands end with ; or \g. Your MySQL connection id is 3221547565 Server version: 5.7.25 OceanBase 3.1.0 (r-00672c3c730c3df6eef3b359eae548d8c2db5ea2) (Built Jun 29 2021 02:14:01) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> use test; Database changed MySQL [test]> create tablegroup tg partition by hash partitions 3; Query OK, 0 rows affected (0.059 sec) MySQL [test]> create table dhytest1(id int , name char(20)); Query OK, 0 rows affected (0.122 sec) MySQL [test]> create table dhytest2(id int , name char(20)) tablegroup=tg partition by hash (id) partitions 3 ; Query OK, 0 rows affected (0.122 sec) MySQL [test]> create table dhytest3(id int , name char(20)) tablegroup=tg partition by hash(id) partitions 3 ; Query OK, 0 rows affected (0.137 sec) MySQL [test]> create table dhytest4(id int , name char(20)) partition by hash (id) partitions 6 ; Query OK, 0 rows affected (0.153 sec)
查看数据分布
这时所有的表 leader 都在 2882 端口上，并且 dhytest4 有 6 个分区，每个 observer 上面分布了两个 leader。
SELECT t.tenant_id, a.tenant_name, t.table_name, d.database_name, tg.tablegroup_name , t.part_num , t2.partition_id, t2.role , t2.ZONE, t2.svr_ip, concat(t2.svr_ip, ':', t2.svr_port) observer , t2.data_size , a.primary_zone , IF(t.locality = '' OR t.locality IS NULL, a.locality, t.locality) AS locality FROM oceanbase.__all_tenant AS a JOIN oceanbase.__all_virtual_database AS d ON ( a.tenant_id = d.tenant_id ) JOIN oceanbase.__all_virtual_table AS t ON (t.tenant_id = d.tenant_id AND t.database_id = d.database_id) JOIN oceanbase.__all_virtual_meta_table t2 ON (t.tenant_id = t2.tenant_id AND (t.table_id=t2.table_id OR t.tablegroup_id=t2.table_id) AND t2.ROLE IN (1) ) LEFT JOIN oceanbase.__all_virtual_tablegroup AS tg ON (t.tenant_id = tg.tenant_id and t.tablegroup_id = tg.tablegroup_id) WHERE a.tenant_id IN (1001,1002 ) AND t.table_type IN (3) and table_name in ('dhytest1','dhytest2','dhytest3', 'dhytest4') ORDER BY t.tenant_id, tg.tablegroup_name, d.database_name, t.table_name, t2.partition_id;
服务器扩容(1-1-1 -> 2-2-2)
设置环境变量和创建目录（三台机器均执行）。
export LD_LIBRARY_PATH=/root/.obd/repository/oceanbase-ce-libs/3.1.0/799bcbe7d5d1dba6eb251b8aca44394bfc5e3c14/ mkdir -p /mydata/observer{2,3}/store/{sort_dir,sstable,clog,ilog,slog}
手动启动 observer 进程。
cd /mydata/observer2 10.140.114.12: /root/.obd/repository/oceanbase-ce/3.1.0/3052acd41b981aed85ce858174e90f07d804f85b/bin/observer -r "10.140.114.12:2882:2881;10.140.60.14:2882:2881;10.140.118.7:2882:2881" -o __min_full_resource_pool_memory=268435456,memory_limit=14G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_disk_percentage=20,enable_syslog_recycle=True,max_syslog_file_count=4 -z zone1 -p 3881 -P 3882 -n ob_cluster -c 1 -d /mydata/observer2/store -i eth0 -l INFO 10.140.118.7: /root/.obd/repository/oceanbase-ce/3.1.0/3052acd41b981aed85ce858174e90f07d804f85b/bin/observer -r "10.140.114.12:2882:2881;10.140.60.14:2882:2881;10.140.118.7:2882:2881" -o __min_full_resource_pool_memory=268435456,memory_limit=14G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_disk_percentage=20,enable_syslog_recycle=True,max_syslog_file_count=4 -z zone2 -p 3881 -P 3882 -n ob_cluster -c 1 -d /mydata/observer2/store -i eth0 -l INFO 10.140.60.14: /root/.obd/repository/oceanbase-ce/3.1.0/3052acd41b981aed85ce858174e90f07d804f85b/bin/observer -r "10.140.114.12:2882:2881;10.140.60.14:2882:2881;10.140.118.7:2882:2881" -o __min_full_resource_pool_memory=268435456,memory_limit=14G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_disk_percentage=20,enable_syslog_recycle=True,max_syslog_file_count=4 -z zone3 -p 3881 -P 3882 -n ob_cluster -c 1 -d /mydata/observer2/store -i eth0 -l INFO
添加 server。
MySQL [oceanbase]> MySQL [oceanbase]> alter system add server '10.140.114.12:3882' zone 'zone1'; Query OK, 0 rows affected (0.055 sec) MySQL [oceanbase]> alter system add server '10.140.60.14:3882' zone 'zone2'; Query OK, 0 rows affected (0.079 sec) MySQL [oceanbase]> alter system add server '10.140.118.7:3882' zone 'zone3'; Query OK, 0 rows affected (0.009 sec)
查看资源情况，现在每个 zone 内已经有两个 observer 了。
select zone,concat(svr_ip,':',svr_port) observer, cpu_capacity,cpu_total,cpu_assigned,cpu_assigned_percent, mem_capacity,mem_total,mem_assigned,mem_assigned_percent, unit_Num,round(`load`,2) `load`, round(cpu_weight,2) cpu_weight, round(memory_weight,2) mem_weight, leader_count from __all_virtual_server_stat order by zone,svr_ip;
租户扩容
升级资源的 unit 数量。
MySQL [oceanbase]> alter resource pool pool_2 unit_num=2; Query OK, 0 rows affected (0.146 sec)
扩容后查看数据分布，dhytest4 这个表中的三个分区的 leader 已经迁移到 3882 上：
SELECT t.tenant_id, a.tenant_name, t.table_name, d.database_name, tg.tablegroup_name , t.part_num , t2.partition_id, t2.role , t2.ZONE, t2.svr_ip, concat(t2.svr_ip, ':', t2.svr_port) observer , t2.data_size , a.primary_zone , IF(t.locality = '' OR t.locality IS NULL, a.locality, t.locality) AS locality FROM oceanbase.__all_tenant AS a JOIN oceanbase.__all_virtual_database AS d ON ( a.tenant_id = d.tenant_id ) JOIN oceanbase.__all_virtual_table AS t ON (t.tenant_id = d.tenant_id AND t.database_id = d.database_id) JOIN oceanbase.__all_virtual_meta_table t2 ON (t.tenant_id = t2.tenant_id AND (t.table_id=t2.table_id OR t.tablegroup_id=t2.table_id) AND t2.ROLE IN (1) ) LEFT JOIN oceanbase.__all_virtual_tablegroup AS tg ON (t.tenant_id = tg.tenant_id and t.tablegroup_id = tg.tablegroup_id) WHERE a.tenant_id IN (1001,1002 ) AND t.table_type IN (3) and table_name in ('dhytest1','dhytest2','dhytest3', 'dhytest4') ORDER BY t.tenant_id, tg.tablegroup_name, d.database_name, t.table_name, t2.partition_id;
租户缩容
MySQL [oceanbase]> alter resource pool pool_2 unit_num=1; Query OK, 0 rows affected (0.146 sec)
查看 __all_rootservice_job 可查看进度，状态从等待变成 SUCCESS。
删除 server。
执行删除 server 后，通过 __all_rootservice_job，__all_virtual_sys_task_status 可查看数据迁移的进度， 这里是因为缩容后数据从 2882 迁移到了 3882，如果删除完 server 后，会再进行一次迁移。
注意：这里也有个问题，扩容时 __all_virtual_sys_task_status 表中查看不到信息。
如果您有任何疑问，可以通过以下方式与我们进行交流：
微信群：扫码添加小助手，将拉你进群哟～
钉钉群：33254054
