今天发现进程挂掉，查看report说是磁盘满，再看一下磁盘空间发现已经清理了，启动该进程，之后进程又挂掉了。
排查过程如下：
1、查看report报告
2023-12-22 11:59:00 INFO OGG-01851 filecaching started: thread ID: 140399863658240.
2023-12-22 11:59:00 INFO OGG-01815 Virtual Memory Facilities for: COM
anon alloc: mmap(MAP_ANON) anon free: munmap
file alloc: mmap(MAP_SHARED) file free: munmap
target directories:
/u01/ogg/dirtmp.
Database Version:
MySQL
Server Version: 5.7.17-log
Client Version: 5.6.14
Host Connection: Localhost via UNIX socket
Protocol Version: 10
Database Language and Character Set:
CLIENT CHARACTER SET = "utf8mb4"
SERVER CHARACTER SET = "utf8mb4"
DATABASE CHARACTER SET = "utf8mb4"
LOCALE INFORMATION = "en"
DATABASE COLLATION = "caseinsensitive"
2023-12-22 11:59:03 INFO OGG-01056 Recovery initialization completed for target file /u01/ogg/dirdat/exp/mo000000032, at RBA 134394099, CSN 000000000000000001785:00000
0559079734.
2023-12-22 11:59:03 INFO OGG-01478 Output file /u01/ogg/dirdat/exp/mo is using format RELEASE 12.3.
2023-12-22 11:59:03 INFO OGG-01026 Rolling over remote file /u01/ogg/dirdat/exp/mo000000032.
2023-12-22 11:59:03 INFO OGG-00182 VAM API running in single-threaded mode.
2023-12-22 11:59:03 INFO OGG-01513 Positioning to Log Number: 1785
Record Offset: 558921462.
INFO !! using index file /data/dbdata/binary-log.index
Source Context :
SourceModule : [ggvam.gen]
SourceID : [/scratch/aime/adestore/views/aime_adc4150325/oggcore/OpenSys/src/gglib/ggvam/cvamgen.cpp]
SourceMethod : [vamInitialize]
SourceLine : [543]
ThreadBacktrace : [14] elements
: [/lib64/libc.so.6(__libc_start_main)]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**))]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**)]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**)]
: [/u01/ogg/extract()]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**)]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**))]
: [/u01/ogg/extract()]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**)]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*))]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**))]
: [/u01/ogg/extract(main)]
: [/lib64/libc.so.6(__libc_start_main)]
: [/u01/ogg/extract()]
2023-12-22 11:59:04 ERROR OGG-00146 Call to VAMInitialize returned with error status 600: VAM Client Report .
2023-12-22 11:59:04 ERROR OGG-01668 PROCESS ABENDING.
根据日志报错信息，发现是日志缺失，但是没有显示是哪个日志
2、查看mysql日志
查看mysql保留的日志情况
mysql> show binary logs;
+-------------------+------------+
| Log_name | File_size |
+-------------------+------------+
| binary-log.001787 | 1079559914 |
| binary-log.001788 | 1082219273 |
| binary-log.001789 | 1074907793 |
| binary-log.001790 | 201447013 |
+-------------------+------------+
查看当前目录binlog日志
srwxrwxrwx 1 mysql mysql 0 Dec 14 17:08 mysql.sock
-rw-r----- 1 mysql mysql 1079559914 Dec 18 05:22 binary-log.001787
-rw-r----- 1 mysql mysql 1082219273 Dec 20 05:28 binary-log.001788
-rw-r----- 1 mysql mysql 1074907793 Dec 22 05:24 binary-log.001789
-rw-r----- 1 mysql mysql 124 Dec 22 05:24 binary-log.index
查看发现日志保留最早的是12.18号的日志。
3、abend进程
GGSCI (bejrun-n-22025 DBLOGIN as ogg) 8> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
EXTRACT ABENDED EXP_1 00:00:01 203:24:32
4、处理
从进程挂起情况来看已经挂起了200多小时，而mysql的日志保留时间只有五天，所以进程挂起。找到问题原因好了，重新初始化进程，进程恢复正常
