Shivam Singhal
About

Looking into Redis Persistence

August 06, 2022

My curiosity around Redis Persistence started when a colleague mentioned something about Redis AOF file and how GCP’s MemoryStore didn’t have support for them.

As usual, I didn’t know what AOF even meant 😛. I got curious and one thing led to another and here we are!

Setup

I installed Redis via Homebrew brew install redis

Redis Server & CLI version is v7.0.3

TLDR

Redis has two modes of data persistence, both of which have their own advantages and disadvantages according to your data needs & application behaviour.

The two modes are:

  1. Redis Dump: A file which contains all the redis’s in-memory data in a binary format.
  2. AOF file: Append-only file: All the commands which the redis clients send are written in multiple file(s) in human readable format.

This post is not about comprehensive difference between Dumps and AOFs. There are official docs for that. This post mainly answers what the dump and the AOFs look like.

Let’s dive:

RDB:

Note: These observations are with default redis config.

Redis regularly dumps all the data in the memory to a dump.rdb file present at /opt/homebrew/var/db/redis

When we kill the redis-server, Redis will dump all the in-memory keys(haven’t tried out the hash buckets yet but I am guessing it dumps those too) into a .rdb file.

When we cat this .rdb file, we can see the key value pairs, although not in the specific order in which we inserted or in a proper format. It is a binary file and that’s why you will also see some unrecognised chars in your terminal.

For AOF:

Note: I had to pass a modified redis.conf file to redis-server  with the option appendonly set to yes . That config file can be found here.*

This is not as straightforward as just taking a RDB dump. It is not a single type of file but three types of file:

  1. A base dump file: This is just an empty or existing .rdb dump which Redis uses as a base to reconstruct the data in combination with the following files.
  2. appendonly.aof.1.incr.aof : This is the actual file which will store the Redis commands we execute (IN HUMAN READABLE FORMAT!). This type of files can be multiple since Redis splits the commands across various files but they all end with .incr.aof
  3. appendonly.aof.manifest : This file contains the metadata about all files of type 1 & type 2^. This manifest file is used by redis-server to reconstruct the data.

All the three files mentioned above can be found here: https://github.com/shhivam/public/tree/main/redis-persistence

If you want to play too, then the relevant config keys in redis.conf file are appendonly, appendfilename, appenddirname

Bonus fact about AOF files

When the AOF files grow too large, Redis “rewrites” the multiple commands with same keys into one single command to save space.

👋

Thanks for reading till here! This will be hopefully one of the many posts about Redis!