部署规划
这一节介绍 OceanBase 集群三节点部署方法，需要通过中控机直接远程登录到 OceanBase 节点上部署启动 observer 和 obproxy 进程。
- 机器信息如下：
- 机器划分如下：
OceanBase 命令行客户端
- 磁盘划分
磁盘划分，这里就使用 LVM 技术对 /dev/vdb 进行划分。需要登录到每个节点上手动初始化。
# lvm 分盘 pvcreate /dev/vdb vgcreate obvg /dev/vdb lvcreate obvg -L 20G^C lvcreate -L 20G obvg -n lvredo lvcreate -l 100%FREE obvg -n lvdata # 格式化文件系统 mkfs.ext4 /dev/obvg/lvdata mkfs.ext4 /dev/obvg/lvredo # 修改 mount 参数文件 vim /etc/fstab /dev/obvg/lvredo /redo ext4 defaults,noatime,nodiratime,nodelalloc,barrier=0 0 0 /dev/obvg/lvdata /data ext4 defaults,noatime,nodiratime,nodelalloc,barrier=0 0 0 # 挂载文件系统 mkdir -p /data /redo vim /etc/fstab mount /data mount /redo chown -R admin.admin /data /redo # 检查 df -h 输出： 文件系统 容量 已用 可用 已用% 挂载点 /dev/mapper/obvg-lvdata 59G 53M 56G 1% /data /dev/mapper/obvg-lvredo 20G 45M 19G 1% /redo
编辑 OBD 配置文件
OBD 针对不同的部署场景提供不同的配置文件。这些配置文件示例在 OceanBase 开源项目地址里，具体是：https://github.com/oceanbase/obdeploy/tree/master/example 。
如果是部署单节点版本，就下载其中两个配置文件：
- 部署三节点
observer进程： https://github.com/oceanbase/obdeploy/blob/master/example/mini-distributed-example.yaml - 部署三节点
observer和obproxy进程：https://github.com/oceanbase/obdeploy/blob/master/example/mini-distributed-with-obproxy-example.yaml
这里仿照生产环境，选择第二种部署配置文件。
[admin@obce00 ~]$ cat obce-3zones.yaml # Only need to configure when remote login is required user: username: admin # password: your password if need key_file: /home/admin/.ssh/id_rsa.pub port: your ssh port, default 22 # timeout: ssh connection timeout (second), default 30 oceanbase-ce: servers: - name: obce01 # Please don't use hostname, only IP can be supported ip: 172.20.249.52 - name: obce02 ip: 172.20.249.49 - name: obce03 ip: 172.20.249.51 global: # Please set devname as the network adaptor's name whose ip is in the setting of severs. # if set severs as "127.0.0.1", please set devname as "lo" # if current ip is 192.168.1.10, and the ip's network adaptor's name is "eth0", please use "eth0" devname: eth0 cluster_id: 2 # please set memory limit to a suitable value which is matching resource. memory_limit: 8G # The maximum running memory for an observer system_memory: 3G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G. stack_size: 512K cpu_count: 16 cache_wash_threshold: 1G __min_full_resource_pool_memory: 268435456 workers_per_cpu_quota: 10 schema_history_expire_time: 1d # The value of net_thread_count had better be same as cpu's core number. net_thread_count: 4 major_freeze_duty_time: Disable minor_freeze_times: 10 enable_separate_sys_clog: 0 enable_merge_by_turn: FALSE #datafile_disk_percentage: 20 # The percentage of the data_dir space to the total disk space. This value takes effect only when datafile_size is 0. The default value is 90. datafile_size: 50G syslog_level: WARN # System log level. The default value is INFO. enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true. enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false. max_syslog_file_count: 10 # The maximum number of reserved log files before enabling auto recycling. The default value is 0. # observer cluster name, consistent with obproxy's cluster_name appname: obce-3zones root_password: 0EI5N08d # root user password, can be empty proxyro_password: uY7Yf8zx # proxyro user pasword, consistent with obproxy's observer_sys_password, can be empty obce01: mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. # The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field. home_path: /home/admin/oceanbase-ce # The directory for data storage. The default value is $home_path/store. data_dir: /data # The directory for clog, ilog, and slog. The default value is the same as the data_dir value. redo_dir: /redo zone: zone1 obce02: mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. # The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field. home_path: /home/admin/oceanbase-ce # The directory for data storage. The default value is $home_path/store. data_dir: /data # The directory for clog, ilog, and slog. The default value is the same as the data_dir value. redo_dir: /redo zone: zone2 obce03: mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. # The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field. home_path: /home/admin/oceanbase-ce # The directory for data storage. The default value is $home_path/store. data_dir: /data # The directory for clog, ilog, and slog. The default value is the same as the data_dir value. redo_dir: /redo zone: zone3 obproxy: servers: - 172.20.249.52 - 172.20.249.49 - 172.20.249.51 # Set dependent components for the component. # When the associated configurations are not done, OBD will automatically get the these configurations from the dependent components. depends: - oceanbase-ce global: listen_port: 2883 # External port. The default value is 2883. prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884. home_path: /home/admin/obproxy # oceanbase root server list # format: ip:mysql_port;ip:mysql_port rs_list: 172.20.249.52:2881;172.20.249.49:2881;172.20.249.51:2881 enable_cluster_checkout: false # observer cluster name, consistent with oceanbase-ce's appname cluster_name: obce-3zones obproxy_sys_password: 0MdTv1tm # obproxy sys user password, can be empty observer_sys_password: uY7Yf8zx # proxyro user pasword, consistent with oceanbase-ce's proxyro_password, can be empty
这个配置文件是专门针对最小内存（可用内存大于 8G）的节点配置，里面指定了很多进程 observer 的启动参数。注意 yaml 的格式，每个配置项后面冒号（:) 跟后面的值之间必须有个空格（' ')。 下面就关键的几个参数补充说明如下：
当上面部署成功后，OBD 会把配置文件 obce-3zones.yaml 复制到自己的工作目录里（~/.obd/cluster/obce-3zones/config.yaml ），后期再改外面这个 obce-3zones.yaml 文件，是不生效的。
OBD 部署集群
配置文件准备好后，就可以部署这个配置文件对应的集群了，部署内容主要包含：
- 复制软件到相应节点，并安装软件。
- 在相应节点创建相关目录。
部署使用命令：obd cluster deploy [集群名] -c 集群配置文件
[admin@obce00 ~]$ obd cluster deploy obce-3zones -c obce-3zones.yaml oceanbase-ce-3.1.0 already installed. obproxy-3.1.0 already installed. +-----------------------------------------------------------------------------+ | Packages | +--------------+---------+---------+------------------------------------------+ | Repository | Version | Release | Md5 | +--------------+---------+---------+------------------------------------------+ | oceanbase-ce | 3.1.0 | 3.el8 | 84bd2fe27f8b8243cc57d8a3f68b4c50f94aab80 | | obproxy | 3.1.0 | 1.el8 | d242ea5fe45222b8f61c3135ba2aaa778c61ea22 | +--------------+---------+---------+------------------------------------------+ Repository integrity check ok Parameter check ok Open ssh connection ok Remote oceanbase-ce-3.1.0-84bd2fe27f8b8243cc57d8a3f68b4c50f94aab80 repository install ok Remote oceanbase-ce-3.1.0-84bd2fe27f8b8243cc57d8a3f68b4c50f94aab80 repository lib check ok Remote obproxy-3.1.0-d242ea5fe45222b8f61c3135ba2aaa778c61ea22 repository install ok Remote obproxy-3.1.0-d242ea5fe45222b8f61c3135ba2aaa778c61ea22 repository lib check ok Cluster status check ok Initializes cluster work home ok Initializes cluster work home ok obce-3zones deployed <p data-line="253" ></p>
检查集群部署状态。
[admin@obce00 ~]$ obd cluster list +----------------------------------------------------------------------+ | Cluster List | +-------------+--------------------------------------+-----------------+ | Name | Configuration Path | Status (Cached) | +-------------+--------------------------------------+-----------------+ | obce-3zones | /home/admin/.obd/cluster/obce-3zones | deployed | +-------------+--------------------------------------+-----------------+ <p data-line="266" ></p>
OBD 启动和初始化集群
上面 deploy 操作只是安装了软件和准备初始化目录，还需要启动集群节点并初始化集群，使用 obd cluster start 命令。
obd cluster start obce-3zones 输出： [admin@obce00 ~]$ obd cluster start obce-3zones Get local repositories and plugins ok Open ssh connection ok Cluster param config check ok Check before start observer ok [WARN] (172.20.249.52) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536) [WARN] (172.20.249.52) The recommended number of open files is 655350 (Current value: 65535) [WARN] (172.20.249.49) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536) [WARN] (172.20.249.49) The recommended number of open files is 655350 (Current value: 65535) [WARN] (172.20.249.51) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536) [WARN] (172.20.249.51) The recommended number of open files is 655350 (Current value: 65535) Check before start obproxy ok Start observer ok observer program health check ok Connect to observer ok Initialize cluster Cluster bootstrap ok Wait for observer init ok +-------------------------------------------------+ | observer | +---------------+---------+------+-------+--------+ | ip | version | port | zone | status | +---------------+---------+------+-------+--------+ | 172.20.249.49 | 3.1.0 | 2881 | zone2 | active | | 172.20.249.51 | 3.1.0 | 2881 | zone3 | active | | 172.20.249.52 | 3.1.0 | 2881 | zone1 | active | +---------------+---------+------+-------+--------+ Start obproxy ok obproxy program health check ok Connect to obproxy ok Initialize cluster +-------------------------------------------------+ | obproxy | +---------------+------+-----------------+--------+ | ip | port | prometheus_port | status | +---------------+------+-----------------+--------+ | 172.20.249.52 | 2883 | 2884 | active | | 172.20.249.49 | 2883 | 2884 | active | | 172.20.249.51 | 2883 | 2884 | active | +---------------+------+-----------------+--------+ obce-3zones running <p data-line="320" ></p>
如果集群节点内核参数和会话限制参数不符合要求，安装会给出提示。 这个命令会在 bootstrap 要几分钟。当可用内存不足 8G 或者日志目录剩余可用空间比例不足 5% 的时候，这个 bootstrap 是很可能会失败的。
接下来进一步确认集群初始化成功。这个步骤是可选的。第一次学习或生产部署的时候，建议检查一下。
- 首先查看启动后的集群状态。
[admin@obce00 ~]$ obd cluster list +----------------------------------------------------------------------+ | Cluster List | +-------------+--------------------------------------+-----------------+ | Name | Configuration Path | Status (Cached) | +-------------+--------------------------------------+-----------------+ | obce-3zones | /home/admin/.obd/cluster/obce-3zones | running | +-------------+--------------------------------------+-----------------+ <p data-line="340" ></p>
- 检查 OceanBase 集群各个节点进程信息。
OceanBase 是单进程软件，进程名叫 observer ，可以用下面命令查看这个进程。
IPS="172.20.249.52 172.20.249.49 172.20.249.51" for ob in $IPS;do echo $ob; ssh $ob "ps -ef | grep observer | grep -v grep "; done 输出： [admin@obce00 oceanbase-ce]$ for ob in $IPS;do echo $ob; ssh $ob "ps -ef | grep observer | grep -v grep "; done 172.20.249.52 admin 6987 1 69 08:35 ? 01:38:26 /home/admin/oceanbase-ce/bin/observer -r 172.20.249.52:2882:2881;172.20.249.49:2882:2881;172.20.249.51:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=3G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=10,root_password=0EI5N08d,redo_dir=/redo -z zone1 -p 2881 -P 2882 -n obce-3zones -c 2 -d /data -i eth0 -l WARN 172.20.249.49 admin 7064 1 87 08:35 ? 02:02:59 /home/admin/oceanbase-ce/bin/observer -r 172.20.249.52:2882:2881;172.20.249.49:2882:2881;172.20.249.51:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=3G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=10,root_password=0EI5N08d,redo_dir=/redo -z zone2 -p 2881 -P 2882 -n obce-3zones -c 2 -d /data -i eth0 -l WARN 172.20.249.51 admin 6920 1 72 08:35 ? 01:42:42 /home/admin/oceanbase-ce/bin/observer -r 172.20.249.52:2882:2881;172.20.249.49:2882:2881;172.20.249.51:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=3G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=10,root_password=0EI5N08d,redo_dir=/redo -z zone3 -p 2881 -P 2882 -n obce-3zones -c 2 -d /data -i eth0 -l WARN <p data-line="358" ></p>
从进程里看，可执行文件是 /home/admin/oceanbase-ce/bin/observer ，实际上它是个软链接。
[admin@obce00 oceanbase-ce]$ ll /home/admin/oceanbase-ce/bin/observer lrwxrwxrwx 1 admin admin 100 Sep 11 17:16 /home/admin/oceanbase-ce/bin/observer -> /home/admin/.obd/repository/oceanbase-ce/3.1.0/84bd2fe27f8b8243cc57d8a3f68b4c50f94aab80/bin/observer <p data-line="365" ></p>
进程启动的时候，通过 -o 指定了很多参数，这些参数都是在前面 OBD 集群部署配置文件里指定的。
- 检查 OceanBase 集群各个节点监听状况。
IPS="172.20.249.52 172.20.249.49 172.20.249.51" for ob in $IPS;do echo $ob; ssh $ob "netstat -ntlp"; done 输出： [admin@obce00 ~]$ for ob in $IPS;do echo $ob; ssh $ob "netstat -ntlp"; done 172.20.249.52 (Not all processes could be identified, non-owned process infowill not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:2881 0.0.0.0:* LISTEN 6987/observer tcp 0 0 0.0.0.0:2882 0.0.0.0:* LISTEN 6987/observer tcp 0 0 0.0.0.0:2883 0.0.0.0:* LISTEN 7640/obproxy tcp 0 0 0.0.0.0:2884 0.0.0.0:* LISTEN 7640/obproxy 172.20.249.49 (Not all processes could be identified, non-owned process infowill not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:2881 0.0.0.0:* LISTEN 7064/observer tcp 0 0 0.0.0.0:2882 0.0.0.0:* LISTEN 7064/observer tcp 0 0 0.0.0.0:2883 0.0.0.0:* LISTEN 7718/obproxy tcp 0 0 0.0.0.0:2884 0.0.0.0:* LISTEN 7718/obproxy tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN - 172.20.249.51 (Not all processes could be identified, non-owned process infowill not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:2881 0.0.0.0:* LISTEN 6920/observer tcp 0 0 0.0.0.0:2882 0.0.0.0:* LISTEN 6920/observer tcp 0 0 0.0.0.0:2883 0.0.0.0:* LISTEN 7574/obproxy tcp 0 0 0.0.0.0:2884 0.0.0.0:* LISTEN 7574/obproxy tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN - <p data-line="407" ></p>
连接 OceanBase 集群的内部实例（sys)
传统的 mysql 客户端可以连接 OceanBase 社区版，前提是 mysql 的版本是 5.5/5.6/5.7 。OceanBase 也提供自己的客户端工具 obclient 需要安装使用。 跟传统 MySQL 不一样的地方是 OBPROXY 的连接端口是 2883 ， 连接用户名是 ：root@sys#集群名 ，密码是前面 OBD 配置文件里指定的。
mysql -h 172.20.249.52 -uroot@sys#obce-3zones -P2883 -p0EI5N08d -c -A oceanbase 输出： [admin@obce00 ~]$ mysql -h 172.20.249.52 -uroot@sys#obce-3zones -P2883 -p0EI5N08d -c -A oceanbase Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 4 Server version: 5.6.25 OceanBase 3.1.0 (r3-b20901e8c84d3ea774beeaca963c67d7802e4b4e) (Built Aug 10 2021 08:10:38) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [oceanbase]> show databases; +--------------------+ | Database | +--------------------+ | oceanbase | | information_schema | | mysql | | SYS | | LBACSYS | | ORAAUDITOR | | test | +--------------------+ 7 rows in set (0.002 sec) MySQL [oceanbase]> select a.zone,concat(a.svr_ip,':',a.svr_port) observer, cpu_total, (cpu_total-cpu_assigned) cpu_free, round(mem_total/1024/1024/1024) mem_total_gb, round((mem_total-mem_assigned)/1024/1024/1024) mem_free_gb, usec_to_time(b.last_offline_time) last_offline_time, usec_to_time(b.start_service_time) start_service_time, b.status, usec_to_time(b.stop_time) stop_time, b.build_version from __all_virtual_server_stat a join __all_server b on (a.svr_ip=b.svr_ip and a.svr_port=b.svr_port) order by a.zone, a.svr_ip ; +-------+--------------------+-----------+----------+--------------+-------------+----------------------------+----------------------------+--------+----------------------------+------------------------------------------------------------------------+ | zone | observer | cpu_total | cpu_free | mem_total_gb | mem_free_gb | last_offline_time | start_service_time | status | stop_time | build_version | +-------+--------------------+-----------+----------+--------------+-------------+----------------------------+----------------------------+--------+----------------------------+------------------------------------------------------------------------+ | zone1 | 172.20.249.52:2882 | 14 | 11.5 | 5 | 4 | 1970-01-01 08:00:00.000000 | 2021-09-12 08:36:06.357140 | active | 1970-01-01 08:00:00.000000 | 3.1.0_3-b20901e8c84d3ea774beeaca963c67d7802e4b4e(Aug 10 2021 08:10:38) | | zone2 | 172.20.249.49:2882 | 14 | 11.5 | 5 | 4 | 1970-01-01 08:00:00.000000 | 2021-09-12 08:36:07.605244 | active | 1970-01-01 08:00:00.000000 | 3.1.0_3-b20901e8c84d3ea774beeaca963c67d7802e4b4e(Aug 10 2021 08:10:38) | | zone3 | 172.20.249.51:2882 | 14 | 11.5 | 5 | 4 | 1970-01-01 08:00:00.000000 | 2021-09-12 08:36:07.631981 | active | 1970-01-01 08:00:00.000000 | 3.1.0_3-b20901e8c84d3ea774beeaca963c67d7802e4b4e(Aug 10 2021 08:10:38) | +-------+--------------------+-----------+----------+--------------+-------------+----------------------------+----------------------------+--------+----------------------------+------------------------------------------------------------------------+ 3 rows in set (0.004 sec) <p data-line="454" ></p>
在数据库列表里看到 oceanbase 这个数据库，就表示集群初始化成功。
obclient 安装和使用示例。
sudo rpm -ivh /tmp/obd/obclient-2.0.0-2.el8.x86_64.rpm /tmp/obd/libobclient-2.0.0-2.el8.x86_64.rpm obclient -h 172.20.249.52 -uroot@sys#obce-3zones -P2883 -p0EI5N08d -c -A oceanbase <p data-line="465" ></p>
附录：
- 第二章 2.1 OceanBase 部署准备
- 第二章 2.2 如何快速体验 OceanBase
- 第二章 2.3 如何规划部署 OceanBase 集群
- 第二章 2.4 如何初始化服务器环境
- 第二章 2.5 安装 OBD 自动化部署软件
- 第二章 2.6 使用 OBD 自动化部署单节点集群
- 第二章 2.7 使用OBD自动化部署多节点集群
- 第二章 2.8 参看和修改 OceanBase 参数集群
- 第二章 2.9 如何部署 OBAgent
- 第二章 2.10 如何重启 OceanBase 集群
- 第二章 2.11（高级）如何手动部署 OceanBase 集群
- 第二章 2.12 常见问题
- 第二章 2.13 附录
结束语
OceanBase社区版入门到实战教程直播正在进行中~
快和小伙伴一起进群交流学习吧~
加入直播群方式一：
钉钉群号3255 4020
加入直播群方式二：
扫码下方二维码加入
