ASM扩容有人直接扩了现有的LUN，来看看怎么搞定
0.起因
群里小伙伴求助，原计划给ASM扩容，结果有人直接把一个LUN在存储端给扩容了。
大佬给出解决方案
我也没干过这种场景，让我来动手模拟下实验试试。
1.准备环境我这的基础实验环境
Oracle linux 6.9 2台 安装Oracle 11.2.0.4 RAC
Oracle linux 6.9 1台安装tgt模拟ISCSI共享存储
现有VOTE磁盘组3块3G[sdc-e]、DATA磁盘组1块12G[sdb]
为了模拟实际效果，我再往DATA磁盘组里加两块盘sdf、sdg大小都12G，使磁盘组变成36G。
扩容完DATA，我再把中间的磁盘SDF改成15G，再尝试扩容ASM磁盘组。
如无问题，扩容完DATA磁盘组变成39G。
创建新存储卷
在TGT存储服务器新建2个卷
[root@tgt-server ~]# lvcreate -L 12g -n data02 oravg
Logical volume "data02" created.
[root@tgt-server ~]# lvcreate -L 12g -n data03 oravg
Logical volume "data03" created.
显示LV
[root@tgt-server ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_root VolGroup -wi-ao---- 111.70g
lv_swap VolGroup -wi-ao---- 7.81g
data oravg -wi-ao---- 12.00g
data02 oravg -wi-a----- 12.00g
data03 oravg -wi-a----- 12.00g
vote01 oravg -wi-ao---- 3.00g
vote02 oravg -wi-ao---- 3.00g
vote03 oravg -wi-ao---- 3.00g
添加映射给RAC节点
[root@tgt-server oravg]# tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 5 --backing-store /dev/oravg/data02
[root@tgt-server oravg]# tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 6 --backing-store /dev/oravg/data03
RAC两节点扫描新存储
RAC1节点
[root@rac1 ~]# iscsiadm -m session --rescan
Rescanning session [sid: 1, target: iqn.2021.tgt-server.control, portal: 192.168.58.100,3260]
[root@rac1 /]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
sda 8:0 0 120G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 119.5G 0 part
├─VolGroup-lv_root (dm-0) 253:0 0 103.5G 0 lvm /
└─VolGroup-lv_swap (dm-1) 253:1 0 16G 0 lvm [SWAP]
sdb 8:16 0 12G 0 disk
sdc 8:32 0 3G 0 disk
sdd 8:48 0 3G 0 disk
sde 8:64 0 3G 0 disk
sdf 8:80 0 12G 0 disk
sdg 8:96 0 12G 0 disk
RAC2节点
[root@rac2 /]# iscsiadm -m session --rescan
Rescanning session [sid: 1, target: iqn.2021.tgt-server.control, portal: 192.168.57.100,3260]
[root@rac2 /]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
sda 8:0 0 120G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 119.5G 0 part
├─VolGroup-lv_root (dm-0) 253:0 0 103.5G 0 lvm /
└─VolGroup-lv_swap (dm-1) 253:1 0 16G 0 lvm [SWAP]
sdb 8:16 0 12G 0 disk
sdc 8:32 0 3G 0 disk
sdd 8:48 0 3G 0 disk
sde 8:64 0 3G 0 disk
sdf 8:80 0 12G 0 disk
sdg 8:96 0 12G 0 disk
生成udev
[root@rac1 ~]# for i in f g ; do echo "KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`", NAME="asm-disk$i", OWNER="grid", GROUP="asmadmin", MODE="0660""; done
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1IET_00010005", NAME="asm-diskf", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1IET_00010006", NAME="asm-diskg", OWNER="grid", GROUP="asmadmin", MODE="0660"
两个节点的配置文件都更新
[root@rac1 ~]# cat /etc/udev/rules.d/80-asm.rules
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1IET_00010001", NAME="asm-diskb", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1IET_00010002", NAME="asm-diskc", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1IET_00010003", NAME="asm-diskd", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1IET_00010004", NAME="asm-diske", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1IET_00010005", NAME="asm-diskf", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1IET_00010006", NAME="asm-diskg", OWNER="grid", GROUP="asmadmin", MODE="0660"
两个节点都执行
[root@rac1 rules.d]# start_udev
Starting udev: [ OK ]
[root@rac2 rules.d]# start_udev
Starting udev: [ OK ]
扩容ASM
登录ASM实例
[grid@rac2 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Mon Apr 1 17:25:00 2024
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> col name for a40;
set line 200;
col path for a30;
select name,path,state,HEADER_STATUS from v$asm_disk;
SQL>
NAME PATH STATE HEADER_STATU
---------------------------------------- ------------------------------ -------- ------------
/dev/asm-diskg NORMAL CANDIDATE
/dev/asm-diskf NORMAL CANDIDATE
VOTE_0001 /dev/asm-diskd NORMAL MEMBER
VOTE_0000 /dev/asm-diskc NORMAL MEMBER
DATA_0000 /dev/asm-diskb NORMAL MEMBER
VOTE_0002 /dev/asm-diske NORMAL MEMBER
SQL> alter diskgroup data add disk '/dev/asm-diskf','/dev/asm-diskg' rebalance power 11;
Diskgroup altered.
SQL> col name for a40;
set line 200;
col path for a30;
select name,path,state,HEADER_STATUS from v$asm_disk;
SQL>
NAME PATH STATE HEADER_STATU
---------------------------------------- ------------------------------ -------- ------------
VOTE_0001 /dev/asm-diskd NORMAL MEMBER
VOTE_0000 /dev/asm-diskc NORMAL MEMBER
DATA_0000 /dev/asm-diskb NORMAL MEMBER
VOTE_0002 /dev/asm-diske NORMAL MEMBER
DATA_0002 /dev/asm-diskg NORMAL MEMBER
DATA_0001 /dev/asm-diskf NORMAL MEMBER
6 rows selected.
SQL> select * from v$asm_operation;
GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
1 REBAL RUN 11 11 343 343 0 0
等待reblance完成
2.准备增加存储扩容asm-diskf磁盘大小
扩容LUN前检查下磁盘使用情况
SQL> SET LINESIZE 200
SET PAGESIZE 100
col groupname format a25 heading 'Disk Group'
col path format a35 heading 'LUN Path'
col diskname format a20 heading 'Disk Name'
col sector_size format 9,999 heading 'Sector |Size'
col block_size format 99,999 heading 'Block |Size'
col state format a10 heading 'State'
col au format 9,999,999 heading 'AU Size'
col total_gig format 999,999 heading 'Group |Total |GB'
col dtotal_gig format 999,999 heading 'Disk |Total |GB'
col free_gig format 999,999 heading 'Group |Free |GB'
col dfree_gig format 999,999 heading 'Disk |Free |GB'
select
g.name groupname,
d.path,
d.name diskname,
d.total_mb/1024 dtotal_gig,
d.free_mb/1024 dfree_gig
from
v$asm_diskgroup g, v$asm_disk d
where
d.group_number = g.group_number
order by
g.name, d.disk_number;SQL> SQL> SQL> SQL> SQL> SQL> SQL> SQL> SQL> SQL> SQL> SQL> SQL> 2 3 4 5 6 7 8 9 10 11 12
Disk Disk
Total Free
Disk Group LUN Path Disk Name GB GB
------------------------- ----------------------------------- -------------------- -------- --------
DATA /dev/asm-diskb DATA_0000 12 11
DATA /dev/asm-diskf DATA_0001 12 11
DATA /dev/asm-diskg DATA_0002 12 11
VOTE /dev/asm-diskc VOTE_0000 3 3
VOTE /dev/asm-diskd VOTE_0001 3 3
VOTE /dev/asm-diske VOTE_0002 3 3
6 rows selected.
SQL>
开始扩展中间的LUN卷大小asm-sdf，由原来的12G扩到15G
[root@tgt-server oravg]# lvresize -L 15G /dev/mapper/oravg-data02
Size of logical volume oravg/data02 changed from 12.00 GiB (3072 extents) to 15.00 GiB (3840 extents).
Logical volume data02 successfully resized.
[root@tgt-server oravg]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_root VolGroup -wi-ao---- 111.70g
lv_swap VolGroup -wi-ao---- 7.81g
data oravg -wi-ao---- 12.00g
data02 oravg -wi-ao---- 15.00g
data03 oravg -wi-ao---- 12.00g
vote01 oravg -wi-ao---- 3.00g
vote02 oravg -wi-ao---- 3.00g
vote03 oravg -wi-ao---- 3.00g
tgt模拟这里直接修改磁盘大小后还要重新映射才能生效，我这里模拟就先正常关集群，再重新映射lv，生产的话，在主机侧重新扫描一下存储就OK，可以看“起因”章节里的截图。
再测试，检查磁盘大小与实际大小
ASM里显示的还是12G，lsblk显示实际盘15G大小。
Total Free
Disk Group LUN Path Disk Name GB GB
------------------------- ----------------------------------- -------------------- -------- --------
DATA /dev/asm-diskb DATA_0000 12 11
DATA /dev/asm-diskf DATA_0001 12 11
DATA /dev/asm-diskg DATA_0002 12 11
VOTE /dev/asm-diskc VOTE_0000 3 3
VOTE /dev/asm-diskd VOTE_0001 3 3
VOTE /dev/asm-diske VOTE_0002 3 3
6 rows selected.
SQL> host lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
sda 8:0 0 120G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 119.5G 0 part
├─VolGroup-lv_root (dm-0) 253:0 0 103.5G 0 lvm /
└─VolGroup-lv_swap (dm-1) 253:1 0 16G 0 lvm [SWAP]
sdb 8:16 0 12G 0 disk
sdc 8:32 0 3G 0 disk
sdd 8:48 0 3G 0 disk
sde 8:64 0 3G 0 disk
sdf 8:80 0 15G 0 disk
sdg 8:96 0 12G 0 disk
扩容ASM
扩容ASM磁盘组
[grid@rac2 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Mon Apr 1 18:24:30 2024
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> select name, total_mb/(1024) "Total GiB" from v$asm_diskgroup;
NAME Total GiB
------------------------------ ----------
DATA 36
VOTE 9
SQL> alter diskgroup data resize all;
Diskgroup altered.
SQL> select name, total_mb/(1024) "Total GiB" from v$asm_diskgroup;
NAME Total GiB
------------------------------ ----------
DATA 39
VOTE 9
查看磁盘使用情况
Disk Disk
Total Free
Disk Group LUN Path Disk Name GB GB
------------------------- ----------------------------------- -------------------- -------- --------
DATA /dev/asm-diskb DATA_0000 12 11
DATA /dev/asm-diskf DATA_0001 15 14
DATA /dev/asm-diskg DATA_0002 12 11
VOTE /dev/asm-diskc VOTE_0000 3 3
VOTE /dev/asm-diskd VOTE_0001 3 3
VOTE /dev/asm-diske VOTE_0002 3 3
6 rows selected.
[grid@rac2 ~]$ asmcmd lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 4194304 39936 37844 0 37844 0 N DATA/
MOUNTED NORMAL N 512 4096 1048576 9216 8290 3072 2609 0 Y VOTE/
调整期间ASM输出日志，期间会自动扩容DATA_0001磁盘，并执行reblance。
NOTE: client orcl2:orcl registered, osid 7534, mbr 0x1
Mon Apr 01 18:24:55 2024
SQL> alter diskgroup data resize all
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=1
NOTE: requesting all-instance disk validation for group=1
Mon Apr 01 18:24:57 2024
NOTE: disk validation pending for group 1/0xedb8fd25 (DATA)
SUCCESS: validated disks for 1/0xedb8fd25 (DATA)
NOTE: increased size in header on grp 1 disk DATA_0001
NOTE: membership refresh pending for group 1/0xedb8fd25 (DATA)
Mon Apr 01 18:25:08 2024
GMON querying group 1 at 9 for pid 18, osid 6281
Mon Apr 01 18:25:08 2024
SUCCESS: refreshed membership for 1/0xedb8fd25 (DATA)
Mon Apr 01 18:25:08 2024
SUCCESS: alter diskgroup data resize all
NOTE: starting rebalance of group 1/0xedb8fd25 (DATA) at power 1
Starting background process ARB0
Mon Apr 01 18:25:08 2024
ARB0 started with pid=28, OS id=8764
NOTE: assigning ARB0 to group 1/0xedb8fd25 (DATA) with 1 parallel I/O
cellip.ora not found.
NOTE: Attempting voting file refresh on diskgroup DATA
NOTE: Refresh completed on diskgroup DATA. No voting file found.
Mon Apr 01 18:25:17 2024
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=1
Mon Apr 01 18:25:20 2024
NOTE: membership refresh pending for group 1/0xedb8fd25 (DATA)
NOTE: stopping process ARB0
SUCCESS: rebalance completed for group 1/0xedb8fd25 (DATA)
Mon Apr 01 18:25:23 2024
GMON querying group 1 at 10 for pid 18, osid 6281
SUCCESS: refreshed membership for 1/0xedb8fd25 (DATA)
NOTE: Attempting voting file refresh on diskgroup DATA
NOTE: Refresh completed on diskgroup DATA. No voting file found.
总结
虽然这样可行，不过还是建议磁盘大小都一样，新增磁盘来扩容，这样规范些。
这么扩了之后，会不会留下啥坑不太确定，不知道以后别的盘满了，这里怎么重分布？？？
参考ACE大佬实验
Resizing Oracle ASM disks - Ron Ekins’ - Oracle Technology, DevOps and Kubernetes Blog
