Redis是一个内存键值对数据库,所以对于内存的管理尤为重要。Redis内部对于内存的管理主要包含两个方向,过期删除策略和数据淘汰策略。思考:
- 什么是数据淘汰?
- 数据过期和数据淘汰都是删除数据,两者有什么区别?
- 实际使用场景是多样化的,如何选择合适的淘汰策略?
淘汰策略原理
所谓数据淘汰是指在Redis内存使用达到一定阈值的时候,执行某种策略释放内存空间,以便于接收新的数据。内存可使用空间由配置参数maxmemory决定(单位mb/GB)。故又叫"最大内存删除策略",也叫"缓存删除策略"。
maxmemory配置
# 客户端命令方式配置和查看内存大小
127.0.0.1:6379> config get maxmemory
"maxmemory"
"0"
127.0.0.1:6379> config set maxmemory 100mb
OK
127.0.0.1:6379> config get maxmemory
"maxmemory"
"104857600"
#通过redis.conf 配置文件配置
127.0.0.1:6379> info
# Server
#...
# 配置文件路径
config_file:/opt/homebrew/etc/redis.conf
#...
# 修改内存大小
> vim /opt/homebrew/etc/redis.conf
############################## MEMORY MANAGEMENT ################################
# Set a memory usage limit to the specified amount of bytes.
# When the memory limit is reached Redis will try to remove keys
# according to the eviction policy selected (see maxmemory-policy).
#
#...
maxmemory 100mb
#...
注:若`maxmemory=0`则表示不做内存限制,但是对于windows系统来说,32位系统默认可使用空间是3G,因为整个系统内存是4G,需要留1G给系统运行。且淘汰策略会自动设置为noeviction,即不开启淘汰策略,当使用空间达到3G的时候,新的内存请求会报错。
淘汰策略分类
- 淘汰策略配置maxmemory-policy,表示当内存达到maxmemory时,将执行配置的淘汰策略,由redis.c/freeMemoryIfNeeded 函数实现数据淘汰逻辑。maxmemory-policy配置
# 命令行配置方式
127.0.0.1:6379> CONFIG GET maxmemory-policy
"maxmemory-policy"
"noeviction"
127.0.0.1:6379> CONFIG SET maxmemory-policy volatile-lru
OK
127.0.0.1:6379> CONFIG GET maxmemory-policy
"maxmemory-policy"
"volatile-lru"
#redis.conf文件配置方式
# MAXMEMORY POLICY: how Redis will select what to remove when maxmemory
# is reached. You can select one from the following behaviors:
#
# volatile-lru -> Evict using approximated LRU, only keys with an expire set.
# allkeys-lru -> Evict any key using approximated LRU.
# volatile-lfu -> Evict using approximated LFU, only keys with an expire set.
# allkeys-lfu -> Evict any key using approximated LFU.
# volatile-random -> Remove a random key having an expire set.
# allkeys-random -> Remove a random key, any key.
# volatile-ttl -> Remove the key with the nearest expire time (minor TTL)
# noeviction -> Don't evict anything, just return an error on write operations.
#
# LRU means Least Recently Used
# LFU means Least Frequently Used
#
# Both LRU, LFU and volatile-ttl are implemented using approximated
# randomized algorithms.
# The default is:
# ...
maxmemory-policy noeviction
freeMemoryIfNeeded逻辑处理
int freeMemoryIfNeeded(void) {
size_t mem_used, mem_tofree, mem_freed;
int slaves = listLength(server.slaves);
/* Remove the size of slaves output buffers and AOF buffer from the count of used memory.*/
// 计算出 Redis 目前占用的内存总数,但有两个方面的内存不会计算在内:
// 1)从服务器的输出缓冲区的内存
// 2)AOF 缓冲区的内存
mem_used = zmalloc_used_memory();
if (slaves) {
listIter li;
listNode *ln;
listRewind(server.slaves,&li);
while((ln = listNext(&li))) {
redisClient *slave = listNodeValue(ln);
unsigned long obuf_bytes = getClientOutputBufferMemoryUsage(slave);
if (obuf_bytes > mem_used)
mem_used = 0;
else
mem_used -= obuf_bytes;
}
}
if (server.aof_state != REDIS_AOF_OFF) {
mem_used -= sdslen(server.aof_buf);
mem_used -= aofRewriteBufferSize();
}
/* Check if we are over the memory limit. */
// 如果目前使用的内存大小比设置的 maxmemory 要小,那么无须执行进一步操作
if (mem_used eviction_pool;
while(bestkey == NULL) {
evictionPoolPopulate(dict, db->dict, db->eviction_pool);
/* Go backward from best to worst element to evict. */
for (k = REDIS_EVICTION_POOL_SIZE-1; k >= 0; k--) {
if (pool[k].key == NULL) continue;
de = dictFind(dict,pool[k].key);
/* Remove the entry from the pool. */
sdsfree(pool[k].key);
/* Shift all elements on its right to left. */
memmove(pool+k,pool+k+1,
sizeof(pool[0])*(REDIS_EVICTION_POOL_SIZE-k-1));
/* Clear the element on the right which is empty since we shifted one position to the left. */
pool[REDIS_EVICTION_POOL_SIZE-1].key = NULL;
pool[REDIS_EVICTION_POOL_SIZE-1].idle = 0;
/* If the key exists, is our pick. Otherwise it is a ghost and we need to try the next element. */
if (de) {
bestkey = dictGetKey(de);
break;
} else {
/* Ghost... */
continue;
}
}
}
}
/* volatile-ttl */
// 策略为 volatile-ttl ,从一集 sample 键中选出过期时间距离当前时间最接近的键
else if (server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_TTL) {
for (k = 0; k < server.maxmemory_samples; k++) {
sds thiskey;
long thisval;
de = dictGetRandomKey(dict);
thiskey = dictGetKey(de);
thisval = (long) dictGetVal(de);
/* Expire sooner (minor expire unix timestamp) is better candidate for deletion */
if (bestkey == NULL || thisval id);
decrRefCount(keyobj);
keys_freed++;
/* When the memory to free starts to be big enough, we may */
/* start spending so much time here that is impossible to */
/* deliver data to the slaves fast enough, so we force the */
/* transmission here inside the loop. */
if (slaves) flushSlavesOutputBuffers();
}
}
if (!keys_freed) return REDIS_ERR; /* nothing to free... */
}
return REDIS_OK;
}
8种淘汰策略
- Redis定义的策略常量(version < 4.0)
/* Redis maxmemory strategies */
#define REDIS_MAXMEMORY_VOLATILE_LRU 0
#define REDIS_MAXMEMORY_VOLATILE_TTL 1
#define REDIS_MAXMEMORY_VOLATILE_RANDOM 2
#define REDIS_MAXMEMORY_ALLKEYS_LRU 3
#define REDIS_MAXMEMORY_ALLKEYS_RANDOM 4
#define REDIS_MAXMEMORY_NO_EVICTION 5
#define REDIS_DEFAULT_MAXMEMORY_POLICY REDIS_MAXMEMORY_NO_EVICTION
3.0版本提供6种策略:
4.0以上版本增加两种LFU策略:
volatile-lfu( REDIS_MAXMEMORY_VOLATILE_LFU): Evict using approximated LFU, only keys with an expire set -> 对配置了过期时间的key,淘汰最近使用频率最少的数据。
allkeys-lfu(REDIS_MAXMEMORY_ALLKEYS_LFU): Evict any key using approximated LFU -> 对所有key,淘汰最近使用频率最少的数据。
volatile-lru( REDIS_MAXMEMORY_VOLATILE_LRU): Evict using approximated LRU, only keys with an expire set -> 内存不足时,对所有配置了过期时间的key,淘汰最近最少使用的数据。
allkeys-lru(REDIS_MAXMEMORY_ALLKEYS_LRU): Evict any key using approximated LRU -> 内存不足时,对所有key,淘汰最近最少使用的数据。
volatile-random( REDIS_MAXMEMORY_VOLATILE_RANDOM): Remove a random key having an expire set -> 内存不足时,对所有配置了过期时间的key,淘汰随机数据。
allkeys-random(REDIS_MAXMEMORY_ALLKEYS_RANDOM): Remove a random key, any key -> 内存不足时,对所有key,淘汰随机数据。
volatile-ttl( REDIS_MAXMEMORY_VOLATILE_TTL): Remove the key with the nearest expire time (minor TTL) -> 内存不足时,对所有配置了过期时间的key,淘汰最近将要过期的数据。
noeviction( REDIS_MAXMEMORY_NO_EVICTION): Don't evict anything, just return an error on write operations -> 不开启淘汰策略,在不配置淘汰策略的情况下,maxmemory-policy默认等于该值。内存不足时,会抛出异常,写操作不可用。不同系统存在差异性-具体见⇑
淘汰策略的选择
- 存在冷热数据区别,即意味着访问频率存在较大差异,4.0及以上版本建议选择allkeys-lfu策略,但要设置lfu-decay-time 计数衰减值,一般默认1,这样可避免缓存污染现象;3.0及以下版本建议选择allkeys-lru策略。LFU访问计数衰减配置
# The counter decay time is the time, in minutes, that must elapse in order
# for the key counter to be divided by two (or decremented if it has a value
# less