Redis内存淘汰和过期删除策略原理分析
Redis是一个内存键值对数据库，所以对于内存的管理尤为重要。Redis内部对于内存的管理主要包含两个方向，过期删除策略和数据淘汰策略。思考：
- 什么是数据淘汰？
- 数据过期和数据淘汰都是删除数据，两者有什么区别？
- 实际使用场景是多样化的，如何选择合适的淘汰策略？
淘汰策略原理
所谓数据淘汰是指在Redis内存使用达到一定阈值的时候，执行某种策略释放内存空间，以便于接收新的数据。内存可使用空间由配置参数maxmemory决定(单位mb/GB)。故又叫"最大内存删除策略"，也叫"缓存删除策略"。
maxmemory配置
# 客户端命令方式配置和查看内存大小
127.0.0.1:6379> config get maxmemory
"maxmemory"
"0"
127.0.0.1:6379> config set maxmemory 100mb
OK
127.0.0.1:6379> config get maxmemory
"maxmemory"
"104857600"
#通过redis.conf 配置文件配置
127.0.0.1:6379> info
1. Server
#...
1. 配置文件路径
config_file:/opt/homebrew/etc/redis.conf
#...
1. 修改内存大小
> vim /opt/homebrew/etc/redis.conf
############################## MEMORY MANAGEMENT ################################
1. Set a memory usage limit to the specified amount of bytes.
1. When the memory limit is reached Redis will try to remove keys
1. according to the eviction policy selected (see maxmemory-policy).
1. #...
maxmemory 100mb
#...注：若`maxmemory=0`则表示不做内存限制，但是对于windows系统来说，32位系统默认可使用空间是3G，因为整个系统内存是4G，需要留1G给系统运行。且淘汰策略会自动设置为noeviction，即不开启淘汰策略，当使用空间达到3G的时候，新的内存请求会报错。
淘汰策略分类
- 淘汰策略配置maxmemory-policy，表示当内存达到maxmemory时，将执行配置的淘汰策略，由redis.c/freeMemoryIfNeeded 函数实现数据淘汰逻辑。maxmemory-policy配置
# 命令行配置方式
127.0.0.1:6379> CONFIG GET maxmemory-policy
"maxmemory-policy"
"noeviction"
127.0.0.1:6379> CONFIG SET maxmemory-policy volatile-lru
OK
127.0.0.1:6379> CONFIG GET maxmemory-policy
"maxmemory-policy"
"volatile-lru"
#redis.conf文件配置方式
1. MAXMEMORY POLICY: how Redis will select what to remove when maxmemory
1. is reached. You can select one from the following behaviors:
1. 1. volatile-lru -> Evict using approximated LRU, only keys with an expire set.
1. allkeys-lru -> Evict any key using approximated LRU.
1. volatile-lfu -> Evict using approximated LFU, only keys with an expire set.
1. allkeys-lfu -> Evict any key using approximated LFU.
1. volatile-random -> Remove a random key having an expire set.
1. allkeys-random -> Remove a random key, any key.
1. volatile-ttl -> Remove the key with the nearest expire time (minor TTL)
1. noeviction -> Don't evict anything, just return an error on write operations.
1. 1. LRU means Least Recently Used
1. LFU means Least Frequently Used
1. 1. Both LRU, LFU and volatile-ttl are implemented using approximated
1. randomized algorithms.
1. The default is:
1. ...
maxmemory-policy noevictionfreeMemoryIfNeeded逻辑处理
int freeMemoryIfNeeded(void) {
size_t mem_used, mem_tofree, mem_freed;
int slaves = listLength(server.slaves);
/* Remove the size of slaves output buffers and AOF buffer from the count of used memory.*/
// 计算出 Redis 目前占用的内存总数，但有两个方面的内存不会计算在内：
// 1）从服务器的输出缓冲区的内存
// 2）AOF 缓冲区的内存
mem_used = zmalloc_used_memory();
if (slaves) {
listIter li;
listNode *ln;
listRewind(server.slaves,&li);
while((ln = listNext(&li))) {
redisClient *slave = listNodeValue(ln);
unsigned long obuf_bytes = getClientOutputBufferMemoryUsage(slave);
if (obuf_bytes > mem_used)
mem_used = 0;
else
mem_used -= obuf_bytes;
}
}
if (server.aof_state != REDIS_AOF_OFF) {
mem_used -= sdslen(server.aof_buf);
mem_used -= aofRewriteBufferSize();
}
/* Check if we are over the memory limit. */
// 如果目前使用的内存大小比设置的 maxmemory 要小，那么无须执行进一步操作
if (mem_used eviction_pool;
while(bestkey == NULL) {
evictionPoolPopulate(dict, db->dict, db->eviction_pool);
/* Go backward from best to worst element to evict. */
for (k = REDIS_EVICTION_POOL_SIZE-1; k >= 0; k--) {
if (pool[k].key == NULL) continue;
de = dictFind(dict,pool[k].key);
/* Remove the entry from the pool. */
sdsfree(pool[k].key);
/* Shift all elements on its right to left. */
memmove(pool+k,pool+k+1,
sizeof(pool[0])*(REDIS_EVICTION_POOL_SIZE-k-1));
/* Clear the element on the right which is empty since we shifted one position to the left. */
pool[REDIS_EVICTION_POOL_SIZE-1].key = NULL;
pool[REDIS_EVICTION_POOL_SIZE-1].idle = 0;
/* If the key exists, is our pick. Otherwise it is a ghost and we need to try the next element. */
if (de) {
bestkey = dictGetKey(de);
break;
} else {
/* Ghost... */
continue;
}
}
}
}
/* volatile-ttl */
// 策略为 volatile-ttl ，从一集 sample 键中选出过期时间距离当前时间最接近的键
else if (server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_TTL) {
for (k = 0; k < server.maxmemory_samples; k++) {
sds thiskey;
long thisval;
de = dictGetRandomKey(dict);
thiskey = dictGetKey(de);
thisval = (long) dictGetVal(de);
/* Expire sooner (minor expire unix timestamp) is better candidate for deletion */
if (bestkey == NULL || thisval id);
decrRefCount(keyobj);
keys_freed++;
/* When the memory to free starts to be big enough, we may */
/* start spending so much time here that is impossible to */
/* deliver data to the slaves fast enough, so we force the */
/* transmission here inside the loop. */
if (slaves) flushSlavesOutputBuffers();
}
}
if (!keys_freed) return REDIS_ERR; /* nothing to free... */
}
return REDIS_OK;
}8种淘汰策略
- Redis定义的策略常量(version < 4.0)
/* Redis maxmemory strategies */
#define REDIS_MAXMEMORY_VOLATILE_LRU 0
#define REDIS_MAXMEMORY_VOLATILE_TTL 1
#define REDIS_MAXMEMORY_VOLATILE_RANDOM 2
#define REDIS_MAXMEMORY_ALLKEYS_LRU 3
#define REDIS_MAXMEMORY_ALLKEYS_RANDOM 4
#define REDIS_MAXMEMORY_NO_EVICTION 5
#define REDIS_DEFAULT_MAXMEMORY_POLICY REDIS_MAXMEMORY_NO_EVICTION3.0版本提供6种策略：
4.0以上版本增加两种LFU策略：
volatile-lfu( REDIS_MAXMEMORY_VOLATILE_LFU): Evict using approximated LFU, only keys with an expire set -> 对配置了过期时间的key，淘汰最近使用频率最少的数据。
allkeys-lfu(REDIS_MAXMEMORY_ALLKEYS_LFU): Evict any key using approximated LFU -> 对所有key，淘汰最近使用频率最少的数据。
volatile-lru( REDIS_MAXMEMORY_VOLATILE_LRU): Evict using approximated LRU, only keys with an expire set -> 内存不足时，对所有配置了过期时间的key，淘汰最近最少使用的数据。
allkeys-lru(REDIS_MAXMEMORY_ALLKEYS_LRU): Evict any key using approximated LRU -> 内存不足时，对所有key，淘汰最近最少使用的数据。
volatile-random( REDIS_MAXMEMORY_VOLATILE_RANDOM): Remove a random key having an expire set -> 内存不足时，对所有配置了过期时间的key，淘汰随机数据。
allkeys-random(REDIS_MAXMEMORY_ALLKEYS_RANDOM): Remove a random key, any key -> 内存不足时，对所有key，淘汰随机数据。
volatile-ttl( REDIS_MAXMEMORY_VOLATILE_TTL): Remove the key with the nearest expire time (minor TTL) -> 内存不足时，对所有配置了过期时间的key，淘汰最近将要过期的数据。
noeviction( REDIS_MAXMEMORY_NO_EVICTION): Don't evict anything, just return an error on write operations -> 不开启淘汰策略，在不配置淘汰策略的情况下，maxmemory-policy默认等于该值。内存不足时，会抛出异常，写操作不可用。不同系统存在差异性-具体见⇑
淘汰策略的选择
- 存在冷热数据区别，即意味着访问频率存在较大差异，4.0及以上版本建议选择allkeys-lfu策略，但要设置lfu-decay-time 计数衰减值，一般默认1，这样可避免缓存污染现象；3.0及以下版本建议选择allkeys-lru策略。LFU访问计数衰减配置
# The counter decay time is the time, in minutes, that must elapse in order
1. for the key counter to be divided by two (or decremented if it has a value
1. less
