oceanbase数据库高版本恢复数据到低版本报错

背景

20230603 晚上同事联系我说测试环境需要恢复一版数据，但是恢复报错。

分析过程

1.看下报错。

MySQL [oceanbase]> alter system restore cs from cs6 at 'file:///OBBACKUP/' until '2023-06-03 04:00:00' with 'backup_cluster_name=obcs6&backup_cluster_id=1632654649&pool_list=restore_pool_cs';  

ERROR 1210 (HY000): Invalid argument

2.因为这个报错不明显，只说无效参数，所以逐步排查下可能的问题。

1>检查下resource_pool是否存在，有没有被租户占用。

MySQL [oceanbase]> select   *  from  __all_tenant;
+----------------------------+----------------------------+-----------+-------------+-------------+-----------+--------------+--------+----------------+---------------+-----------+-----------------------+---------------+---------------------+-------------------+------------------------+-----------------------------+-----------------------+--------------------+------------------+----------------------+---------------+
| gmt_create                 | gmt_modified               | tenant_id | tenant_name | replica_num | zone_list | primary_zone | locked | collation_type | info          | read_only | rewrite_merge_version | locality      | logonly_replica_num | previous_locality | storage_format_version | storage_format_work_version | default_tablegroup_id | compatibility_mode | drop_tenant_time | status               | in_recyclebin |
+----------------------------+----------------------------+-----------+-------------+-------------+-----------+--------------+--------+----------------+---------------+-----------+-----------------------+---------------+---------------------+-------------------+------------------------+-----------------------------+-----------------------+--------------------+------------------+----------------------+---------------+
| 2022-07-13 15:10:14.220146 | 2022-07-13 15:10:14.220146 |         1 | sys         |          -1 | zone1     | zone1        |      0 |              0 | system tenant |         0 |                     0 | FULL{1}@zone1 |                   0 |                   |                      0 |                           0 |                    -1 |                  0 |               -1 | TENANT_STATUS_NORMAL |             0 |
+----------------------------+----------------------------+-----------+-------------+-------------+-----------+--------------+--------+----------------+---------------+-----------+-----------------------+---------------+---------------------+-------------------+------------------------+-----------------------------+-----------------------+--------------------+------------------+----------------------+---------------+
1 row in set (0.01 sec)

MySQL [oceanbase]> select  *  from  __all_resource_pool;
+----------------------------+----------------------------+------------------+------------------+------------+----------------+-----------+-----------+--------------+--------------------+
| gmt_create                 | gmt_modified               | resource_pool_id | name             | unit_count | unit_config_id | zone_list | tenant_id | replica_type | is_tenant_sys_pool |
+----------------------------+----------------------------+------------------+------------------+------------+----------------+-----------+-----------+--------------+--------------------+
| 2022-07-13 15:09:30.203458 | 2022-07-13 15:09:30.208544 |                1 | sys_pool         |          1 |              1 | zone1     |         1 |            0 |                  0 |
| 2022-07-19 17:35:02.858501 | 2023-06-03 12:52:50.605821 |             1002 | restore_pool_cs  |          2 |           1002 | zone1     |        -1 |            0 |                  0 |
| 2023-03-01 18:22:12.989484 | 2023-03-01 18:22:12.989484 |             1007 | restore_pool_cs1 |          2 |           1005 | zone1     |        -1 |            0 |                  0 |
+----------------------------+----------------------------+------------------+------------------+------------+----------------+-----------+-----------+--------------+--------------------+
3 rows in set (0.00 sec)

2>检查下nfs，发现是v3挂载，生产该nfs是v4挂载，v4也是OB推荐的nfs版本。

obcs07:/OBBACKUP # ls -ld /OBBACKUP
drwxrwxrwx 14 admin admin 4096 Jun  3 23:00 /OBBACKUP
obcs07:/ # umount /OBBACKUP
umount.nfs: /OBBACKUP: device is busy
obcs07:/ # lsof /OBBACKUP
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
mysql    3850 root  cwd    DIR   0,43     4096   65 /OBBACKUP (111.11.11.1:/CS-OB-BACKUP)
bash    21960 root  cwd    DIR   0,43     4096   65 /OBBACKUP (111.11.11.1:/CS-OB-BACKUP)
obcs07:/ # kill -9 3850
obcs07:/ # kill -9  21960
obcs07:/ # lsof /OBBACKUP
obcs07:/ # umount /OBBACKUP
obcs07:/ # mount -t nfs -o vers=4,minorversion=1,proto=tcp,rsize=1048567,wsize=1048567,hard,rw,sync,intr,timeo=600,lookupcache=positive,namlen=255 111.11.11.1:/CS-OB-BACKUP /OBBACKUP
obcs07:/ # ls -ld /OBBACKUP
drwxrwxrwx 14 nobody nobody 4096 Jun  3 23:00 /OBBACKUP

3>检查下备份源端库的信息和可恢复时间点。

MySQL [oceanbase]> select * from __all_cluster;
+----------------------------+----------------------------+-------------+------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------------------+---------------------+
| gmt_create                 | gmt_modified               | cluster_idx | cluster_id | cluster_name | rs_list                                                                                                                                                                                                                                                                                                                                                                              | cluster_status | redo_transport_options       | protection_level    |
+----------------------------+----------------------------+-------------+------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------------------+---------------------+
| 2022-03-30 10:29:21.700991 | 2023-05-19 00:01:23.177934 |           0 | 1632654649 | obcs6       | {"Data":{"ObRegion":"obcs6","ObCluster":"obcs6","ObRegionId":1632654649,"ObClusterId":1632654649,"Type":"PRIMARY","timestamp":1684425683174294,"RsList":[{"address":"10.19.193.73:2882","role":"LEADER","sql_port":2881},{"address":"10.19.193.14:2882","role":"FOLLOWER","sql_port":2881},{"address":"134.84.21.7:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]}} |              1 | ASYNC NET_TIMEOUT = 30000000 | MAXIMUM PERFORMANCE |
+----------------------------+----------------------------+-------------+------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------------------+---------------------+
1 row in set (0.00 sec)

MySQL [oceanbase]> select * from __all_tenant;
+----------------------------+----------------------------+-----------+-------------+-------------+-------------------+-------------------+--------+----------------+---------------+-----------+-----------------------+---------------------------------------------+---------------------+-------------------+------------------------+-----------------------------+-----------------------+--------------------+------------------+----------------------+---------------+
| gmt_create                 | gmt_modified               | tenant_id | tenant_name | replica_num | zone_list         | primary_zone      | locked | collation_type | info          | read_only | rewrite_merge_version | locality                                    | logonly_replica_num | previous_locality | storage_format_version | storage_format_work_version | default_tablegroup_id | compatibility_mode | drop_tenant_time | status               | in_recyclebin |
+----------------------------+----------------------------+-----------+-------------+-------------+-------------------+-------------------+--------+----------------+---------------+-----------+-----------------------+---------------------------------------------+---------------------+-------------------+------------------------+-----------------------------+-----------------------+--------------------+------------------+----------------------+---------------+
| 2022-07-23 01:04:25.918279 | 2022-07-23 01:04:25.918279 |         1 | sys         |          -1 | zone1;zone2;zone3 | zone2;zone1;zone3 |      0 |              0 | system tenant |         0 |                     0 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |                   0 |                   |                      0 |                           0 |                    -1 |                  0 |               -1 | TENANT_STATUS_NORMAL |             0 |
| 2023-05-03 00:49:57.817803 | 2023-05-03 00:49:57.817803 |      1002 | cs6        |          -1 | zone1;zone2;zone3 | zone3;zone2;zone1 |      0 |              0 |               |         0 |                     0 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |                   0 |                   |                      0 |                           0 |                    -1 |                  1 |               -1 | TENANT_STATUS_NORMAL |             0 |
+----------------------------+----------------------------+-----------+-------------+-------------+-------------------+-------------------+--------+----------------+---------------+-----------+-----------------------+---------------------------------------------+---------------------+-------------------+------------------------+-----------------------------+-----------------------+--------------------+------------------+----------------------+---------------+
2 rows in set (0.00 sec)
MySQL [oceanbase]> select incarnation,tenant_id,bs_key,start_time,completion_time,elapsed_secondes,time_taken_display,status from cdb_ob_backup_set_details order by start_time,tenant_id;
+-------------+-----------+--------+----------------------------+----------------------------+------------------+--------------------+-----------+
| incarnation | tenant_id | bs_key | start_time                 | completion_time            | elapsed_secondes | time_taken_display | status    |
+-------------+-----------+--------+----------------------------+----------------------------+------------------+--------------------+-----------+
|           1 |         1 |     50 | 2023-05-20 04:00:14.854324 | 2023-05-20 04:34:15.743749 |             2041 | 00:34:00.889425    | COMPLETED |
|           1 |      1002 |     50 | 2023-05-20 04:00:14.854324 | 2023-05-20 04:33:42.385832 |             2008 | 00:33:27.531508    | COMPLETED |
|           1 |         1 |     51 | 2023-05-27 04:00:11.804028 | 2023-05-27 04:34:03.128465 |             2031 | 00:33:51.324437    | COMPLETED |
|           1 |      1002 |     51 | 2023-05-27 04:00:11.804028 | 2023-05-27 04:33:35.311909 |             2004 | 00:33:23.507881    | COMPLETED |
|           1 |         1 |     52 | 2023-06-03 04:00:13.185440 | 2023-06-03 04:29:19.185607 |             1746 | 00:29:06.000167    | COMPLETED |
|           1 |      1002 |     52 | 2023-06-03 04:00:13.185440 | 2023-06-03 04:28:53.305023 |             1720 | 00:28:40.119583    | COMPLETED |
+-------------+-----------+--------+----------------------------+----------------------------+------------------+--------------------+-----------+
6 rows in set (0.00 sec)

3.再执行下命令看下。

MySQL [oceanbase]> alter system restore crm from crm6 at 'file:///OBPHYBACKUP' until '2023-06-03 04:40:00' with 'backup_cluster_name=obcrm6&backup_cluster_id=1632654649&pool_list=restore_pool_yy';
ERROR 1210 (HY000): Invalid argument

4.命令确认没问题，环境也检查了，怀疑与版本可能有关系，检查下版本。

恢复集群：3.2.1_20220507193726-fdda6552273a134448b268dc9451eab45efdbf75(May  7 2022 20:14:08)

备份集群：3.2.3.3_107060012023041113-3cfe0f0783ae40aa8a41ec7d074ab7dedb93f702(Apr 11 2023 13:55:51)

官网上是有明确标注的：

在进行恢复操作前，请务必确认待恢复的备份数据的版本，OceanBase 数据库当前仅支持将低版本的备份数据恢复到同版本或高版本中，同版本下的小版本之间也不支持逆向恢复。例如，如果您的备份数据为 OceanBase 数据库 V3.2.1 版本，则仅支持将该数据恢复到 OceanBase 数据库 V3.2.1 及以上版本中。（ https://www.oceanbase.com/docs/enterprise-oceanbase-database-cn-10000000000368207 ）

简单来说，低版本可以恢复数据到高版本，高版本不能恢复到低版本，那么这个问题就很清楚了，升级下测试库版本应该就可以了。

5.升级测试库环境（ps：有个小插曲）。

1>因为比较晚了，我在ocp上发起升级任务就去洗漱了，回来看到进程起来了发现server版本已经升级了。

MySQL [oceanbase]> select  *  from  __all_server;
+----------------------------+----------------------------+--------------+----------+----+-------+------------+-----------------+--------+-----------------------+-------------------------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| gmt_create                 | gmt_modified               | svr_ip       | svr_port | id | zone  | inner_port | with_rootserver | status | block_migrate_in_time | build_version                                                                             | stop_time | start_service_time | first_sessid | with_partition | last_offline_time |
+----------------------------+----------------------------+--------------+----------+----+-------+------------+-----------------+--------+-----------------------+-------------------------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| 2022-07-13 15:09:21.239575 | 2023-06-03 23:42:34.256991 | 133.34.36.13 |     2882 |  1 | zone1 |       2881 |               1 | active |                     0 | 3.2.3.3_107060012023041113-3cfe0f0783ae40aa8a41ec7d074ab7dedb93f702(Apr 11 2023 13:55:51) |         0 |   1685806953267838 |            0 |              1 |                 0 |
| 2022-07-13 15:09:39.626145 | 2023-06-03 23:42:44.269228 | 133.34.36.14 |     2882 |  2 | zone1 |       2881 |               0 | active |                     0 | 3.2.3.3_107060012023041113-3cfe0f0783ae40aa8a41ec7d074ab7dedb93f702(Apr 11 2023 13:55:51) |         0 |   1685806963270715 |            0 |              1 |                 0 |
+----------------------------+----------------------------+--------------+----------+----+-------+------------+-----------------+--------+-----------------------+-------------------------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+

2>那么再次执行下恢复命令吧。

MySQL [oceanbase]> alter system restore cs from cs6 at 'file:///OBBACKUP' until '2023-06-03 04:40:00' with 'backup_cluster_name=obcs6&backup_cluster_id=1632654649&pool_list=restore_pool_cs';
ERROR 4179 (HY000): restore tenant while in standby cluster or in upgrade mode not allowed

3>看报错说升级期间不允许恢复，我以为升级成功但是ocp调用有问题导致的，我手工处理了下（其实我应该先在ocp确认升级任务的）。

MySQL [oceanbase]> select gmt_modified,name,value from __all_sys_parameter where name not in ('obconfig_url','rootservice_list','backup_dest_option');
+----------------------------+-------------------------------+-------------------------------------+
| gmt_modified               | name                          | value                               |
+----------------------------+-------------------------------+-------------------------------------+
| 2022-07-13 15:09:39.658811 | all_server_list               | 133.34.36.13:2882,133.34.36.14:2882 |
| 2023-06-03 23:35:35.381183 | enable_ddl                    | False                               |
| 2022-07-13 15:09:39.362272 | enable_one_phase_commit       | 0                                   |
| 2023-06-03 23:32:40.443175 | enable_sys_table_ddl          | False                               |
| 2023-06-03 23:30:01.358313 | enable_upgrade_mode           | true                                |
| 2023-06-03 23:30:01.358313 | min_observer_version          | 3.2.1                               |
| 2023-06-03 23:29:45.153394 | replica_safe_remove_time      | 72h                                 |
| 2022-07-19 17:40:57.989016 | restore_concurrency           | 100                                 |
| 2023-06-03 23:29:40.091202 | server_permanent_offline_time | 72h                                 |
| 2022-07-13 15:09:39.405602 | _enable_oracle_priv_check     | 1                                   |
| 2023-06-03 23:29:50.223512 | _enable_static_typing_engine  | 0                                   |
| 2022-07-13 15:09:39.314018 | _max_trx_size                 | 100M                                |
| 2022-07-13 15:09:39.448910 | _partition_balance_strategy   | standard                            |
| 2023-06-03 23:30:01.358313 | _upgrade_stage                | PREUPGRADE                          |
+----------------------------+-------------------------------+-------------------------------------+
14 rows in set (0.00 sec)

MySQL [oceanbase]> alter system end upgrade;
Query OK, 0 rows affected (0.05 sec)

4>在ocp上确认升级任务，发现任务在等待observer启动那步超时了，这时候我才意识到是ocp调度还没执行完，先把状态改回去让ocp自己调度。

MySQL [oceanbase]>  alter system  begin  upgrade;
Query OK, 0 rows affected (0.05 sec)

MySQL [oceanbase]> select name,value from __all_sys_parameter where name like '%upgrade%';
+---------------------+-------+
| name                | value |
+---------------------+-------+
| enable_upgrade_mode | false |
| _upgrade_stage      | NONE  |
+---------------------+-------+
2 rows in set (0.01 sec)

5>把ocp任务重试，检查完成后，再执行下命令。

MySQL [oceanbase]> alter system restore cs from cs6 at 'file:///OBBACKUP' until '2023-06-03 04:40:00' with 'backup_cluster_name=obcs6&backup_cluster_id=1632654649&pool_list=restore_pool_cs';
Query OK, 0 rows affected (0.06 sec)

总结

其实处理问题就要丝抽茧剥，先把自己的思路整理清楚，根据报错提示和自己分析的顺序一点点验证和推断，直到拨云见日，问题迎刃而解。

行之所向，莫问远方。