实践练习二（必选）：手动部署 OceanBase 集群
历史实践题回顾
- 实践1：OceanBase Docker 体验
练习内容
- （必选）手动部署一个 OB 单副本集群，包括一个 OBProxy 节点。
- （必选）创建一个业务租户、一个业务数据库，以及一些表等。
- （可选）如果单台服务器内存有32G，或者有三台服务器，改为部署一个 OB 三副本集群，包括一个 OBProxy 节点。
- （可选）如果有三台服务器并且服务器内存有 32 G，可以单服务器内启动 2 个节点，实现 1-1-1 扩容到 2-2-2 。
具体实现
笔记本电脑资源有限，只能部署一个 OB 单副本集群和一个OBProxy 节点的学习环境。
部署一个OB单副本集群（含一个OBProxy）
准备安装介质
社区版地址
- 官网下载
- GitHub 下载
- 阿里云 Yum 源
支持的操作系统
OceanBase 社区版支持的操作系统包括：
- CentOS ：推荐7.2 以后版本。
- Debian ：推荐 9.8， 10.9 版本。
- openSUSE ：推荐 15.2 版本。
- OpenAnolis：推荐 8.2 版本。
- SUSE ： 推荐 15.2 版本。
- Ubuntu：推荐 16.04 、18.04、20.04 等版本
资源需求
- 每个observer 进程需要至少10G内存容量，2个监听端口（默认是 2881 和 2882 ）以及至少10G的数据目录
- obproxy OceanBase 的反向代理进程，可以独立部署在应用服务器，独立的机器或者OceanBase 的机器上。生产环境建议至少部署两个obproxy
- 默认使用admin用户管理OB
初始化服务器环境
调整内核参数
cat > /etc/sysctl.d/98-obce.conf <<-'EOF' net.core.somaxconn = 2048 net.core.netdev_max_backlog = 10000 net.core.rmem_default = 16777216 net.core.wmem_default = 16777216 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.ip_local_port_range = 3500 65535 net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.tcp_syncookies = 0 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_fin_timeout = 15 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_slow_start_after_idle=0 vm.swappiness = 0 vm.min_free_kbytes = 2097152 vm.max_map_count=655360 fs.aio-max-nr=1048576 EOF sysctl --system
调整资源限制
OceanBase 数据库的进程涉及的限制包括线程最大栈空间大小（Stack）、最大文件句柄数（Open Files）和 core 文件大小 （Core File Size）资源
cat > /etc/security/limits.d/98-obce.conf <<-'EOF' # 最大栈空间大小设置为 unlimited，最大文件句柄数设置为 655350，Core 文件大小设置为 unlimited * soft nofile 655360 * hard nofile 655360 * soft nproc 655360 * hard nproc 655360 * soft core unlimited * hard core unlimited * soft stack unlimited * hard stack unlimited EOF # 检查确认 ulimit -a
会话需要退出后重登录才生效
关闭防火墙
systemctl status firewalld systemctl disable --now firewalld
关闭SELinux
setenforce 0 getenforce sed -i 's/=enforcing/=disabled/g' /etc/selinux/config grep '^SELINUX=' /etc/selinux/config
需要重启主机后生效
时间同步
生产环境所有节点间的时间误差建议控制在 10ms 以内。
- 安装 chrony 服务
yum -y install chrony
- 配置 chrony
vi /etc/chrony.conf # server 后面跟时间同步服务器 # 使用pool.ntp.org 项目中的公共服务器。按 server 配置，理论上您想添加多少时间服务器都可以。 # 或者使用 阿里云的 ntp 服务器 # Please consider joining the pool (http://www.pool.ntp.org/join.html). server ntp.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst server ntp.aliyun.com minpoll 4 maxpoll 10 iburst server ntp1.aliyun.com minpoll 4 maxpoll 10 iburst server ntp1.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst server ntp10.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst # 如果是测试环境，没有时间同步服务器，那就选取一台配置为时间同步服务器。 # 如果选中的是本机，则取消下面 server 注释 #server 127.127.1.0 # 根据实际时间计算出服务器增减时间的比率，然后记录到一个文件中，在系统重启后为系统做出最佳时间补偿调整。 driftfile /var/lib/chrony/drift # chronyd 根据需求减慢或加速时间调整， # 在某些情况下系统时钟可能漂移过快，导致时间调整用时过长。 # 该指令强制 chronyd 调整时期，大于某个阀值时步进调整系统时钟。 # 只有在因 chronyd 启动时间超过指定的限制时（可使用负值来禁用限制）没有更多时钟更新时才生效。 makestep 1.0 3 # 将启用一个内核模式，在该模式中，系统时间每11分钟会拷贝到实时时钟（RTC）。 rtcsync # Enable hardware timestamping on all interfaces that support it. # 通过使用hwtimestamp指令启用硬件时间戳 #hwtimestamp eth0 #hwtimestamp eth1 #hwtimestamp * # Increase the minimum number of selectable sources required to adjust # the system clock. #minsources 2 # 指定一台主机、子网，或者网络以允许或拒绝NTP连接到扮演时钟服务器的机器 #allow 192.168.0.0/16 #deny 192.168/16 # 即使没有同步到时间源，也要服务时间 local stratum 10 # 指定包含NTP验证密钥的文件。 #keyfile /etc/chrony.keys # 指定日志文件的目录。 logdir /var/log/chrony # Select which information is logged. #log measurements statistics tracking
- 检查 chrony 状态信息
# 查看时间同步活动 chronyc activity # 查看时间服务器 chronyc sources # 查看同步状态 chronyc sources -v # 校准时间服务器： chronyc tracking # 使用 clockdiff [-o] <host> 命令可以检查本机跟目标机器的时间同步误差
时区设置（可选）
# 查看所有可用时区 timedatectl list-timezones # 设置当前系统时区 timedatectl set-timezone Asia/Shanghai # 强制同步下系统时钟 chronyc -a makestep
创建安装用户
#添加普通用户 groupadd -g 1000 ober useradd -u 1000 -g ober -m -d /ups/app/oceanbase -c "OceanBase Owner" ober #更改密码 echo "ober" | passwd --stdin ober #运行oceanbase可运行任何命令，不需要密码 visudo #添加oceanbase一行内容 ## Same thing without a password # %wheel ALL=(ALL) NOPASSWD: ALL ober ALL=(ALL) NOPASSWD: ALL
配置SSH免密登陆
使用OBD工具部署集群，需要配置中控机的 OBD 运行的用户到 OceanBase 集群节点的 OBSERVER 安装的用户的 SSH 免密登录
export SSH='ssh -o PasswordAuthentication=no -o StrictHostKeyChecking=no'
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/*.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys ober@<nodeN>:~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
# 检查确认
export SSH='ssh -o ConnectTimeout=3 -o ConnectionAttempts=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no'
for hosts in $(grep -Ev "^#|\-vip|scan" /etc/hosts|awk '{print $NF}'); do
${SSH} ${hosts} date
done
准备磁盘文件系统
- 运行日志。在启动目录下的 log 目录里。主要记录进程 observer 的运行日志、选举服务的运行日志和 rootservice 的运行日志。主要读写特点是顺序写。
- 数据文件。主要是指数据文件 block_file ，一次性初始化大小，后面可以在线扩容，但是不能缩容。主要读写特点是随机读、顺序写。偶尔密集的随机写。
- 事务日志文件。主要是指事务和 sstable 相关的日志 ，包括 clog、ilog 和 slog 等。主要读写特点是顺序写。
这三类文件尽可能的分散在不同的磁盘上存储。如果物理上只有一块盘，则可以使用 fdisk 或 lvm 划分为多个逻辑盘。
- 修改 mount 参数文件
# lvm 分盘 pvcreate /dev/sdb vgcreate datavg /dev/sdb lvcreate -L 20G datavg -n lvredo lvcreate -l 100%FREE datavg -n lvdata # 格式化文件系统 mkfs.ext4 /dev/obvg/lvdata mkfs.ext4 /dev/obvg/lvredo # 修改 mount 参数文件 vim /etc/fstab /dev/obvg/lvredo /redo ext4 defaults,noatime,nodiratime,nodelalloc,barrier=0 0 0 /dev/obvg/lvdata /data ext4 defaults,noatime,nodiratime,nodelalloc,barrier=0 0 0 # 挂载文件系统 mount -a chown -R admin.admin /data /redo
安装OBD工具
OBD 全称是 OceanBase Deployer，是 OceanBase 社区版的命令行下自动化部署软件。
离线安装 OBD
su - ober # 下载 curl https://mirrors.aliyun.com/oceanbase/community/stable/el/7/x86_64/ob-deploy-1.1.2-1.el7.x86_64.rpm # 安装 sudo rpm -ivh ob-deploy-1.1.2-1.el7.x86_64.rpm # 加载环境变量 source /etc/profile.d/obd.sh # 查看可执行文件的安装路径 rpm -ql `rpm -qa|grep ob-deploy` # OBD 工作文件的目录： ~/.obd/ tree ~/.obd -L 1
- 配置离线仓库
# 1. 首先要删除远程仓库 mv ~/.obd/mirror/remote/OceanBase.repo ~/.obd/mirror/remote/OceanBase.repo.bak # 2. 将所需的软件上传到指定目录，如：/ups/app/obrpm/ # 3. 离线仓库 obd mirror clone /ups/app/obrpm/*.rpm # 4. 输出本地仓库中rpm包列表 obd mirror list local
在线联网安装 OBD
# yum 安装 yum install -y yum-utils yum-config-manager --add-repo https://mirrors.aliyun.com/oceanbase/OceanBase.repo yum install -y ob-deploy # 查看一下 OceanBase.repo 内容 cat /etc/yum.repos.d/OceanBase.repo # OceanBase.repo [oceanbase.community.stable] name=OceanBase-community-stable-el$releasever baseurl=http://mirrors.aliyun.com/oceanbase/community/stable/el/$releasever/$basearch/ enabled=1 gpgcheck=1 gpgkey=http://mirrors.aliyun.com/oceanbase/RPM-GPG-KEY-OceanBase [oceanbase.development-kit] name=OceanBase-development-kit-el$releasever baseurl=http://mirrors.aliyun.com/oceanbase/development-kit/el/$releasever/$basearch/ enabled=1 gpgcheck=1 gpgkey=http://mirrors.aliyun.com/oceanbase/RPM-GPG-KEY-OceanBase
obd工具的用法
]$ obd -h Usage: obd <command> [options] Available commands: cluster Deploy and manage a cluster. mirror Manage a component repository for OBD. repo Manage local repository for OBD. test Run test for a running deploy deployment. update Update OBD. Options: --version show program's version number and exit -h, --help Show help and exit. -v, --verbose Activate verbose output.
编辑OBD配置文件
OBD 针对不同的部署场景提供不同的配置文件。
部署单节点 observer 和 obproxy 进程配置文件
## Only need to configure when remote login is required # user: # username: your username # password: your password if need # key_file: your ssh-key file path if need # port: your ssh port, default 22 # timeout: ssh connection timeout (second), default 30 oceanbase-ce:servers:# Please don't use hostname, only IP can be supported- 127.0.0.1global:# The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.home_path: /home/admin/oceanbase-ce# The directory for data storage. The default value is $home_path/store.# data_dir: /data# The directory for clog, ilog, and slog. The default value is the same as the data_dir value.# redo_dir: /redo# Please set devname as the network adaptor's name whose ip is in the setting of severs.# if set severs as "127.0.0.1", please set devname as "lo"# if current ip is 192.168.1.10, and the ip's network adaptor's name is "eth0", please use "eth0"devname: bond0mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.zone: zone1cluster_id: 1# please set memory limit to a suitable value which is matching resource. memory_limit: 8G # The maximum running memory for an observersystem_memory: 4G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.stack_size: 512Kcpu_count: 16cache_wash_threshold: 1G__min_full_resource_pool_memory: 268435456workers_per_cpu_quota: 10schema_history_expire_time: 1d# The value of net_thread_count had better be same as cpu's core number. net_thread_count: 4major_freeze_duty_time: Disableminor_freeze_times: 10enable_separate_sys_clog: 0enable_merge_by_turn: FALSEdatafile_disk_percentage: 20 # The percentage of the data_dir space to the total disk space. This value takes effect only when datafile_size is 0. The default value is 90.syslog_level: WARN # System log level. The default value is INFO.enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.max_syslog_file_count: 6 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.# observer cluster name, consistent with obproxy's cluster_nameappname: obcluster# root_password: # root user password, can be empty# proxyro_password: # proxyro user pasword, consistent with obproxy's observer_sys_password, can be empty obproxy:# Set dependent components for the component.# When the associated configurations are not done, OBD will automatically get the these configurations from the dependent components.depends:- oceanbase-ceservers:- 127.0.0.1global:listen_port: 2883 # External port. The default value is 2883.prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.home_path: /root/obproxy# oceanbase root server list# format: ip:mysql_port;ip:mysql_port. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.# rs_list: 192.168.1.2:2881;192.168.1.3:2881;192.168.1.4:2881rs_list: 127.0.0.1:2881enable_cluster_checkout: false# observer cluster name, consistent with oceanbase-ce's appname. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.# cluster_name: obclusterskip_proxy_sys_private_check: true# obproxy_sys_password: # obproxy sys user password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.# observer_sys_password: # proxyro user pasword, consistent with oceanbase-ce's proxyro_password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
部署三节点 observer 和 obproxy 进程
## Only need to configure when remote login is required # user: # username: your username # password: your password if need # key_file: your ssh-key file path if need # port: your ssh port, default 22 # timeout: ssh connection timeout (second), default 30 oceanbase-ce:servers:- name: server1# Please don't use hostname, only IP can be supportedip: 192.168.1.2- name: server2ip: 192.168.1.3- name: server3ip: 192.168.1.4global:# Please set devname as the network adaptor's name whose ip is in the setting of severs.# if set severs as "127.0.0.1", please set devname as "lo"# if current ip is 192.168.1.10, and the ip's network adaptor's name is "eth0", please use "eth0"devname: eth0cluster_id: 1# please set memory limit to a suitable value which is matching resource. memory_limit: 8G # The maximum running memory for an observersystem_memory: 4G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.stack_size: 512Kcpu_count: 16cache_wash_threshold: 1G__min_full_resource_pool_memory: 268435456workers_per_cpu_quota: 10schema_history_expire_time: 1d# The value of net_thread_count had better be same as cpu's core number. net_thread_count: 4major_freeze_duty_time: Disableminor_freeze_times: 10enable_separate_sys_clog: 0enable_merge_by_turn: FALSEdatafile_disk_percentage: 20 # The percentage of the data_dir space to the total disk space. This value takes effect only when datafile_size is 0. The default value is 90.syslog_level: INFO # System log level. The default value is INFO.enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.max_syslog_file_count: 4 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.# observer cluster name, consistent with obproxy's cluster_nameappname: obcluster# root_password: # root user password, can be empty# proxyro_password: # proxyro user pasword, consistent with obproxy's observer_sys_password, can be emptyserver1:mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.# The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.home_path: /root/observer# The directory for data storage. The default value is $home_path/store.# data_dir: /data# The directory for clog, ilog, and slog. The default value is the same as the data_dir value.# redo_dir: /redozone: zone1server2:mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.# The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.home_path: /root/observer# The directory for data storage. The default value is $home_path/store.# data_dir: /data# The directory for clog, ilog, and slog. The default value is the same as the data_dir value.# redo_dir: /redozone: zone2server3:mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.# The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.home_path: /root/observer# The directory for data storage. The default value is $home_path/store.# data_dir: /data# The directory for clog, ilog, and slog. The default value is the same as the data_dir value.# redo_dir: /redozone: zone3 obproxy:# Set dependent components for the component.# When the associated configurations are not done, OBD will automatically get the these configurations from the dependent components.depends:- oceanbase-ceservers:- 192.168.1.5global:listen_port: 2883 # External port. The default value is 2883.prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.home_path: /root/obproxy# oceanbase root server list# format: ip:mysql_port;ip:mysql_port. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.# rs_list: 192.168.1.2:2881;192.168.1.3:2881;192.168.1.4:2881enable_cluster_checkout: false# observer cluster name, consistent with oceanbase-ce's appname. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.# cluster_name: obclusterskip_proxy_sys_private_check: true# obproxy_sys_password: # obproxy sys user password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.# observer_sys_password: # proxyro user pasword, consistent with oceanbase-ce's proxyro_password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
参数项说明
当上面部署成功后，OBD 会把配置文件 obce-single.yaml 复制到自己的工作目录里（~/.obd/cluster/obce-single/config.yaml ），后期再改外面这个 obce-single.yaml 文件，是不生效的。
部署集群
deploy 操作只是安装了软件和准备初始化目录
# 用法：obd cluster deploy [集群名] -c 集群配置文件 obd cluster deploy obce-single -c obce-single.yaml
检查集群状态
obd cluster list
启动和初始化集群
obd cluster start obce-single
处理：
编辑obd cluster edit-config obce配置文件，添加rs_list参数项配置后重启即可或检查depends项内容是否正确
集群状态
obd工具
obd cluster list obd cluster display obce-single # 检查数据文件大小 ls -lrth /data/sstable/block_file
observer 进程启动后会初始化数据文件（block_file）大小，根据参数 datafile_size 或 datafile_disk_percentage 控制。
OS工具
- ps 工具
# 查看进程 ps aux |grep observer # 查看对应的线程 ps -T -p $(pidof observer) ps -L -o pid,lwp,pri,nice,start,stat,bsdtime,cmd,comm -C ob
- top 工具
# top -H -p <pid> top -H -p $(pidof observer)
连接OceanBase 集群的内部实例（sys)
安装客户端工具obclient
rpm -ivh obclient-2.0.0-2.el7.x86_64.rpm libobclient-2.0.0-2.el7.x86_64.rpm
连接OB
# 1.1 通过 MySQL 5.5/5.6/5.7连接 mysql -h 127.0.0.1 -uroot@sys -P2881 -c -A oceanbase # 1.2 通过obclient工具连接 obclient -h 127.0.0.1 -uroot@sys -P2881 -c -A oceanbase # 查看集群配置相关信息 select a.zone,concat(a.svr_ip,':',a.svr_port) observer, cpu_total, (cpu_total-cpu_assigned) cpu_free, round(mem_total/1024/1024/1024) mem_total_gb, round((mem_total-mem_assigned)/1024/1024/1024) mem_free_gb, usec_to_time(b.last_offline_time) last_offline_time, usec_to_time(b.start_service_time) start_service_time, b.status, usec_to_time(b.stop_time) stop_time, b.build_version from __all_virtual_server_stat a join __all_server b on (a.svr_ip=b.svr_ip and a.svr_port=b.svr_port) order by a.zone, a.svr_ip ; show databases;
创建业务数据
创建mysql租户
# 登陆
## obclient -h127.1 -uroot@sys -P2881 -c -A oceanbase
obclient -h127.1 -uroot@sys#obce-single -P2883 -prootPWD123 -c -A oceanbase
show databases;
alter resource unit sys_unit_config min_cpu=2;
# 创建资源单元
CREATE resource unit S2C1G max_cpu=2, min_cpu=2, max_memory='1G', min_memory='1G', max_iops=10000, min_iops=1000, max_session_num=10000, max_disk_size='10G';
# 创建资源池
CREATE resource pool my_pool unit = 'S2C1G', unit_num = 1;
# 创建租户obmysql
create tenant obmysql resource_pool_list=('my_pool'), primary_zone='RANDOM',comment 'mysql tenant/instance', charset='utf8' set ob_tcp_invited_nodes='%', ob_compatibility_mode='mysql';
exit;
创建数据库
# 使用新建的租户登陆并创建业务数据库db1 obclient -h 127.1 -uroot@obmysql#obce-single -P2883 -c -A test create database db1;
创建表
use db1; create table t1(id int) ; insert into t1 values (1),(2); select * from t1;
附录
https://open.oceanbase.com/answer/detail?id=13700696&sou=0a001
参考资料
- 社区版官网-文档-学习中心-入门教程：如何手动部署 OceanBase 集群
- 社区版官网-博客-入门实战：如何手动部署 OceanBase 集群
- 教程视频：【2-4-如何手动部署OceanBase三副本集群.mp4]
