redis哨兵与VIP-每日运维

通常我们在配置redis哨兵的时候，对于应用。要么通过配置中心来管理redis的连接IP，要么通过配置文件来配置。再或者通过负载均衡的拆解来配置，以达到后端节点发生变化后的适配。现在我们使用另外一种办法，VIP飘逸的方式来简单实现。目的是在后端主从发生变化后，不需要修改任何配置文件。

首先安装

yum install gcc-c++ -y
curl -Lk https://download.redis.io/releases/redis-5.0.9.tar.gz|tar xz -C /usr/local/
ln -s /usr/local/redis-5.0.9 /usr/local/redis
cd /usr/local/redis-5.0.9/ && make && make install

而后最简化配置

cd /usr/local/redis-5.0.9
egrep -v "^$|^#" redis.conf > /etc/redis.conf
sed -i 's/bind 127.0.0.1/bind 0.0.0.0/g'  /etc/redis.conf
sed -i 's/protected-mode yes/protected-mode no/g' /etc/redis.conf
sed -i 's/daemonize no/daemonize yes/g'  /etc/redis.conf
sed -i 's@logfile ""@logfile "/data/redis/redis.log"@g'  /etc/redis.conf
sed -i 's@dir ./@dir /data/redis@g' /etc/redis.conf
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo "requirepass password" >> /etc/redis.conf

创建用户

groupadd redis
useradd redis -s /sbin/nologin -g redis
mkdir -p /data/redis
chown -R redis.redis /data/redis

创建启动脚本

cat > /etc/systemd/system/redis.service <<EOF
[Unit]
Description=Redis
After=network.target
LimitNOFILE=infinity
[Service]
User=redis
Group=redis
Type=forking
Type=forking
ExecStart=/usr/local/bin/redis-server /etc/redis.conf
ExecStop=/usr/local/bin/redis-cli -a password shutdown
PrivateTmp=true

 
[Install]
WantedBy=multi-user.target
EOF

启动后测试密码连通性

systemctl daemon-reload
systemctl enable redis
systemctl start redis.service

[root@master1 redis]# redis-cli -a password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info
# Server
redis_version:5.0.9
...
# Replication
role:master
connected_slaves:0
master_replid:1ab243c0b889ef65c88ca1501cd9eadb7006ffad
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
...
127.0.0.1:6379>

配置从库

将上述命令在另外2台分别操作后，启动另外2台，并且指向151

slaveof 172.16.100.151 6379

redis2

并且我们另外需要在slave添加主库的密码

echo "masterauth password" >> /etc/redis.conf
systemctl stop redis
systemctl start redis

操作如下

[root@master2 redis-5.0.9]# redis-cli -a password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> slaveof 172.16.100.151 6379
OK
127.0.0.1:6379> info
...
# Replication
role:slave
master_host:172.16.100.151
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:392
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:c08f4025e08009774421b8d08ed33e17222ec3aa
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:392
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:393
repl_backlog_histlen:0
...

redis3

并且我们另外需要在slave添加主库的密码

echo "masterauth password" >> /etc/redis.conf
systemctl stop redis
systemctl start redis

开始配置

[root@master3 redis-5.0.9]# redis-cli -a password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> slaveof 172.16.100.151 6379
OK
127.0.0.1:6379> info
....
# Replication
role:slave
master_host:172.16.100.151
master_port:6379
master_link_status:up
master_last_io_seconds_ago:3
master_sync_in_progress:0
slave_repl_offset:336
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:5f7eeb3456ecb98e91e8ab9882aa41adddf81210
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:336
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:295
repl_backlog_histlen:42
....

主从测试

此时主从已经完成

[root@master1 ~]# redis-cli -a password 
127.0.0.1:6379> info 
...
# Replication
role:master
connected_slaves:2
slave0:ip=172.16.100.153,port=6379,state=online,offset=420,lag=0
slave1:ip=172.16.100.152,port=6379,state=online,offset=420,lag=1
master_replid:c08f4025e08009774421b8d08ed33e17222ec3aa
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:420
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:420
...

在redis1的master上set数据会同步到redis2和redis3的slave

[root@master1 ~]# redis-cli -a password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> set name linuxea.com
OK
127.0.0.1:6379> get name
"linuxea.com"

redis2

[root@master2 manifests]# redis-cli -a password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> get name
"linuxea.com"

redis3

[root@master2 manifests]# redis-cli -a password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> get name
"linuxea.com"

sentinel

cd /usr/local/redis-5.0.9
egrep -v "^$|^#" sentinel.conf

分别在三台节点添加如下

cat > /etc/redis-sentinel.conf << EOF
port 26379
daemonize yes
pidfile /var/run/redis-sentinel.pid
logfile "/data/redis/redis-sentinel.log"
dir /tmp
sentinel monitor mymaster 172.16.100.151 6379 2
sentinel down-after-milliseconds mymaster 10
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 30
sentinel deny-scripts-reconfig yes
EOF

cat > /etc/systemd/system/redis-sentinel.service <<EOF
[Unit]
Description=Redis
After=network.target
LimitNOFILE=infinity

[Service]
User=redis
Group=redis
Type=forking
Type=forking
ExecStart=/usr/local/bin/redis-sentinel /etc/redis-sentinel.conf
ExecStop=/usr/local/bin/redis-sentinel  shutdown
PrivateTmp=true

 
[Install]
WantedBy=multi-user.target
EOF

chown redis.redis /etc/redis-sentinel.conf
systemctl daemon-reload
systemctl enable redis-sentinel
systemctl start redis-sentinel.service

现在每台节点的master都指向172.16.100.151:6379

[root@master2 ~]# redis-cli -a password -p 26379 info
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=sdown,address=172.16.100.151:6379,slaves=0,sentinels=1

模拟故障

关闭redis1

[root@master1 ~]# redis-cli -a password -p 6379 shutdown
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

通过观察日志查看master的飘逸

27090:X 15 Nov 2023 01:05:55.253 # +new-epoch 8
27090:X 15 Nov 2023 01:05:55.253 # +try-failover master mymaster 172.16.100.152 6379
27090:X 15 Nov 2023 01:05:55.321 # +vote-for-leader cca1e3c4c7a7ad5785ea5df8ef1f2491d74765a6 8
27090:X 15 Nov 2023 01:05:55.321 # +sdown slave 172.16.100.153:6379 172.16.100.153 6379 @ mymaster 172.16.100.152 6379
27090:X 15 Nov 2023 01:05:55.321 # +sdown slave 172.16.100.151:6379 172.16.100.151 6379 @ mymaster 172.16.100.152 6379
27090:X 15 Nov 2023 01:05:55.321 # +sdown sentinel bb5414fec115afcb651cf52a2a000df8b7a9e236 172.16.100.153 26379 @ mymaster 172.16.100.152 6379
27090:X 15 Nov 2023 01:05:55.321 # +sdown sentinel a9eb32c3559d786ed7fd9fad5207980c1c978ec9 172.16.100.151 26379 @ mymaster 172.16.100.152 6379
27090:X 15 Nov 2023 01:05:55.321 # bb5414fec115afcb651cf52a2a000df8b7a9e236 voted for bb5414fec115afcb651cf52a2a000df8b7a9e236 8
27090:X 15 Nov 2023 01:05:55.321 # a9eb32c3559d786ed7fd9fad5207980c1c978ec9 voted for a9eb32c3559d786ed7fd9fad5207980c1c978ec9 8
27090:X 15 Nov 2023 01:05:55.476 # +new-epoch 9
27090:X 15 Nov 2023 01:05:55.543 # +vote-for-leader a9eb32c3559d786ed7fd9fad5207980c1c978ec9 9
27090:X 15 Nov 2023 01:05:55.543 # -sdown slave 172.16.100.153:6379 172.16.100.153 6379 @ mymaster 172.16.100.152 6379
27090:X 15 Nov 2023 01:05:55.543 # -sdown sentinel bb5414fec115afcb651cf52a2a000df8b7a9e236 172.16.100.153 26379 @ mymaster 172.16.100.152 6379
27090:X 15 Nov 2023 01:05:55.543 # -sdown sentinel a9eb32c3559d786ed7fd9fad5207980c1c978ec9 172.16.100.151 26379 @ mymaster 172.16.100.152 6379
27090:X 15 Nov 2023 01:05:55.792 # -sdown master mymaster 172.16.100.152 6379
27090:X 15 Nov 2023 01:05:55.792 # -odown master mymaster 172.16.100.152 6379
27090:X 15 Nov 2023 01:05:55.792 # +sdown slave 172.16.100.153:6379 172.16.100.153 6379 @ mymaster 172.16.100.152 6379
27090:X 15 Nov 2023 01:05:56.089 # -failover-abort-not-elected master mymaster 172.16.100.152 6379
27090:X 15 Nov 2023 01:05:56.366 # -sdown slave 172.16.100.153:6379 172.16.100.153 6379 @ mymaster 172.16.100.152 6379

redis2

[root@master2 ~]# redis-cli -a password info

# Replication
role:master
connected_slaves:1
slave0:ip=172.16.100.153,port=6379,state=online,offset=22085,lag=1
master_replid:edadf6db3ac41b966941a232bbe99a238a04168e
master_replid2:428e166444dcad9f182fe6c916ca9cf19e6393b9
master_repl_offset:22085
second_repl_offset:15148
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:22085

如下

[root@master2 ~]# redis-cli -a password -p 26379 info

# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=172.16.100.152:6379,slaves=2,sentinels=3

redis3

[root@master3 ~]# redis-cli -a password -p 6379 info
...
# Replication
role:slave
master_host:172.16.100.152
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:40787
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:edadf6db3ac41b966941a232bbe99a238a04168e
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:40787
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:18174
repl_backlog_histlen:22614
...

如下

[root@master3 ~]# redis-cli -a password -p 26379 info
...
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=172.16.100.152:6379,slaves=2,sentinels=3
...

redis-vip

我们希望模拟Master故障，能够通过一个VIP来自动的转移，而不需要去修改代码或者人为的过早介入，那么我们需要配置一个VIP来探测

#!/bin/bash
################################
#判断当前节点是否为master
# by marksugar www.linuxea.com
# 如果不是就删除VIP，如果是就添加VIP
################################
. /etc/profile
. ~/.bash_profile
vip=172.16.100.199
device=eth0
ss -tlnp|grep :6379 || { ip addr del $vip dev $device;echo "redis is not runing `date +%F-%T`">> /var/log/redis-vip.log;exit -1; }
chown redis.redis /etc/redis-sentinel.conf
if [ `echo $(/usr/local/bin/redis-cli -a password --no-auth-warning info | awk -F: '/role/{print $2}')|grep slave|wc -l` -eq 1 ]; then
  echo "is slave `date +%F-%T`" >> /var/log/redis-vip.log
  if [ `/usr/sbin/ip a|grep $vip|wc -l` -eq 1 ]; then
    ip addr del $vip dev $device
    echo "is slave,now del $device $vip `date +%F-%T`" >> /var/log/redis-vip.log   
  else 
    echo "is slave,not vip $vip `date +%F-%T`" >> /var/log/redis-vip.log
  fi
else
  echo "is master `date +%F-%T`" >> /var/log/redis-vip.log
  if [ `/usr/sbin/ip a|grep $vip|wc -l` -eq 1 ]; then
    echo "is master,VIP exists $device $vip `date +%F-%T`" >> /var/log/redis-vip.log
  else 
    echo "is master, not vip $vip `date +%F-%T`,now configure $vip dev $device" >>  /var/log/redis-vip.log
    /usr/sbin/ip addr add $vip dev $device
  fi
fi

而后这样

* * * * * sleep 10; /root/vip.sh
* * * * * sleep 20; /root/vip.sh
* * * * * sleep 30; /root/vip.sh
* * * * * sleep 40; /root/vip.sh
* * * * * sleep 50; /root/vip.sh

此时在进行手动关闭Master

[root@master2 ~]# redis-cli -a password shutdown
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
[root@master2 ~]#

当前节点会剔除VIP

is master 2023-11-15-19:04:41
is master,VIP exists eth0 172.16.100.199 2023-11-15-19:04:41
is master 2023-11-15-19:04:51
is master,VIP exists eth0 172.16.100.199 2023-11-15-19:04:51
redis is not runing 2023-11-15-19:05:11
redis is not runing 2023-11-15-19:05:21
redis is not runing 2023-11-15-19:02:11
redis is not runing 2023-11-15-19:02:21
redis is not runing 2023-11-15-19:02:31
redis is not runing 2023-11-15-19:02:41
redis is not runing 2023-11-15-19:02:51
redis is not runing 2023-11-15-19:03:11
redis is not runing 2023-11-15-19:03:21
redis is not runing 2023-11-15-19:03:31

而slave节点会接任VIP

is slave 2023-11-15-19:04:41
is slave,not vip 172.16.100.199 2023-11-15-19:04:41
is slave 2023-11-15-19:04:51
is slave,not vip 172.16.100.199 2023-11-15-19:04:51
is master 2023-11-15-19:05:11
is master, not vip 172.16.100.199 2023-11-15-19:05:11,now configure 172.16.100.199 dev eth0
is master 2023-11-15-19:05:21
is master,VIP exists eth0 172.16.100.199 2023-11-15-19:05:21
is master 2023-11-15-19:05:31
is master,VIP exists eth0 172.16.100.199 2023-11-15-19:05:31
is master 2023-11-15-19:05:41
is master,VIP exists eth0 172.16.100.199 2023-11-15-19:05:41

关闭的节点恢复

redis is not runing 2023-11-15-19:03:21
redis is not runing 2023-11-15-19:03:31
is master 2023-11-15-19:03:41
is master, not vip 172.16.100.199 2023-11-15-19:03:41,now configure 172.16.100.199 dev eth0
is slave 2023-11-15-19:03:51
is slave,now del eth0 172.16.100.199 2023-11-15-19:03:51
is slave 2023-11-15-19:04:11
is slave,not vip 172.16.100.199 2023-11-15-19:04:11

redis哨兵与VIP

配置从库

redis2

redis3

主从测试

sentinel

模拟故障

redis-vip

相关文章

发布评论取消回复

向阳逐梦

【Rocky 9】Step by Step 从 CentOS 7.9 升级到 Rocky Linux 9.2

oracle查询V$ARCHIVE_GAP慢的问题解决–转瞻Derek

Oracle 11g RAC集群日常运维命令总结

redis哨兵与VIP

配置从库

redis2

redis3

主从测试

sentinel

模拟故障

redis-vip

相关文章

发布评论 取消回复

向阳逐梦

【Rocky 9】Step by Step 从 CentOS 7.9 升级到 Rocky Linux 9.2

oracle查询V$ARCHIVE_GAP慢的问题解决–转瞻Derek

Oracle 11g RAC集群日常运维命令总结

发布评论取消回复