Redis, from the Ground Up（3）

Expiry

The EXPIRE command enables a timeout to be set for a specified key. Once the timeout is reached, the server can delete the key. We call a key volatile if it is set to expire.

Note that expiry semantics changed very significantly with Redis 2.1.3. It is important to be aware of the differences, especially given the wide install base of Redis 2.0.

Redis Versions Prior to 2.1.3

The initial design of Redis' expiry mechanism is extremely concerned with consistency.

Here, if the value of a volatile key is modified in any way, then the key becomes non-volatile and will not expire unless explicitly EXPIRE’d again.

Furthermore, the following restrictions apply:

• If a key is volatile, the EXPIRE command cannot be re-issued to change the expiry time
• A write operation against a volatile key will destroy the key first and thenperform the write

For example:

    redis> SET foo bar
    OK
    redis> EXPIRE foo 100000
    (integer) 1
    redis> GET foo
    "bar"
    redis> APPEND foo 123
    (integer) 3
    redis> GET foo
    "123"

In order to enforce consistency requirements, this is expected behaviour. (To be strongly consistent, if n servers receive the same list of commands in the same sequence, they must always end with the same dataset in memory. However, with replication over unreliable network links, and without the restrictions detailed above, slowdowns in the network could cause the master and slaves to get out sync, thus violating the consistency requirement.)

Redis Versions 2.1.3 and Above

None of these constraints apply to the later versions of Redis (i.e., write operations no longer destroy the key before performing the write, and expiry times can be modified after initially being set). The implementation was changed to remove the constraints,without sacrificing the consistency requirements.

The new implementation simply injects a DEL (delete) operation into the replication stream and the AOF file every time that the server expires a key.

More on Strings

We previously demonstrated the GET, SET, and APPEND commands.

Increment

The INCR command makes it possible to use strings as an atomic counter.

For example:

    redis> SET foo 0
    OK
    redis> INCR foo
    (integer) 1
    redis> INCR foo
    (integer) 2
    redis> GET foo
    "2"
    redis> SET bar baz
    OK
    redis> INCR bar
    (error) ERR value is not an integer

SETNX and Locking

The SETNX (“set if not exists”) command works like SET, but performs no operation if the key already exists.

This command can be used to implement a locking system. For example, the Redis documentation describes a locking algorithm.

Lists

Lists are used to store an (ordered) collection of items. Stacks and queues can be very easily modelled with lists.

For example:

    redis> LPUSH newset a
    (integer) 1
    redis> LRANGE newset 0 -1
    1. "a"
    redis> LPUSH newset b
    (integer) 2
    redis> RPUSH newset c
    (integer) 3
    redis> LRANGE newset 0 -1
    1. "b"
    2. "a"
    3. "c"
    redis> LPUSH myset z
    (integer) 1
    redis> RPOPLPUSH newset myset
    "c"
    redis> LRANGE newset 0 -1
    1. "b"
    2. "a"
    redis> LRANGE myset 0 -1
    1. "c"
    2. "z"

Sets

Sets store an un-ordered, unique collection of items. In addition to supporting sets, Redis has native support for for union, intersection, and diff operations.

    redis> SADD set1 a
    (integer) 1
    redis> SADD set1 b
    (integer) 1
    redis> SADD set1 c
    (integer) 1
    redis> SADD set2 c
    (integer) 1
    redis> SADD set2 d
    (integer) 1
    redis> SADD set2 e
    (integer) 1
    redis> SINTER set1 set2
    1. "c"
    redis> SUNION set1 set2
    1. "c"
    2. "d"
    3. "a"
    4. "b"
    5. "e"

Sets are a critical tool for building highly-optimized indexes. For example, if your application allows users to like articles and follow users, then you should be manually maintaining indexes (i.e. sets) to pre-compute the answers to questions like “which users liked article x” and “which users follow user y”.

Sorted Sets

Sorted sets (also known as “zsets”) are similar to sets, but each member of the set has an associated floating point score. Members are stored in sorted order, so no sorting is required to retrieve the data ordered by the floating point score.

For example:

    redis> ZADD days 1 Monday
    (integer) 1
    redis> ZADD days 2 Tuesday
    (integer) 1
    redis> ZADD days 3 Wednesday
    (integer) 1
    redis> ZADD days 4 Thursday
    (integer) 1
    redis> ZADD days 5 Friday
    (integer) 1
    redis> ZADD days 6 Saturday
    (integer) 1
    redis> ZADD days 7 Sunday
    (integer) 1
    redis> ZRANGE days 0 -1
    1. "Monday"
    2. "Tuesday"
    3. "Wednesday"
    4. "Thursday"
    5. "Friday"
    6. "Saturday"
    7. "Sunday"

Sorted sets are essential whenever a range query is needed. For a clever application of sorted sets, see Auto Complete with Redis.

Furthermore, as with plain sets, sorted sets are an important tool to consider when constructing indexes.

Hashes

Redis hashes are conceptually equivalent to the Ruby hash, Python dictionary, Java hash table or hash map, etc.

For example:

    redis> HSET user:1 name bob
    (integer) 1
    redis> HSET user:1 email b@bob.com
    (integer) 1
    redis> HGET user:1 name
    "bob"
    redis> HGETALL user:1
    1. "name"
    2. "bob"
    3. "email"
    4. "[email protected]"

Before this data type was added to Redis, users had two main options for storing hashes: use one key to store a JSON structure with multiple fields, or use one key for each field in the hash. Both of these approaches have many downsides versus using the native hash data type; the former introduces concurrency issues, and the latter requires significantly more memory.

Redis Transactions

Redis transactions can be used to make a series of commands atomic. (Every Redis command is already atomic; transactions, however, allow us to combine a number of commands into a single atomic unit.)

In this example, the SET and two INCR commands are all executed atomically within the context of a transaction:

    redis> MULTI
    OK
    redis> SET foo 0
    QUEUED
    redis> INCR foo
    QUEUED
    redis> INCR foo
    QUEUED
    redis> EXEC
    1. OK
    2. (integer) 1
    3. (integer) 2

Here, MULTI delineates the start of a transaction block; EXEC is responsible for actually running the transaction (i.e., executing all commands between the MULTI and the EXEC, with no other commands executed in-between).

There is an important caveat: because the commands are queued and then executed sequentially (all at once, as if they were a single unit), it is not possible to read a value while a transaction is being executed and to then subsequently use this value at any point during the course of the same transaction.

To address this restriction, Redis supports “Check and Set” (CAS) transactions. CAS is available via the WATCH command.

The following example (in pseudo-code) is an example of a good implementation of a function that only increments a key if the current value of the key is even.

    WATCH mykey 
    val = GET mykey
    MULTI
    if val % 2 == 0:
        SET mykey (val+1)
    EXEC

If mykey changes after the WATCH command is executed but before the EXEC, then the entire transaction is aborted. The client can retry indefinitely on transaction failure. Any implementation that does not not use CAS is subject to race conditions, unless some other locking mechanism is employed (for example, the locking system described during the treatment of SETNX).

A more involved example, taken from an earlier post I made to the Redis mailing list:

I'm looking to build a FIFO queue, with one important/ non-standard property: if an item that already exists in the queue is added again, then the insert should effectively be ignored. (This data structure can be useful for managing certain types of jobs, like warming caches, etc.) 

An efficient way to model this structure in Redis would be to use a ZSET, where the score of each item in the set is the unix timestamp of when the item was added. 

However, the semantics of ZADD are that if an item already exists in the set, that the score is updated to the score specified in the command. If there were some way to tell ZADD to not update the score if an element already exists, or perhaps to supply a certain score if the element is new and a different score if the element is not new, then this queue-like data structure could be easily (and very efficiently) modelled in Redis. 

There are potentially a few other ways to do this, with the obvious approach being to check if an item already exists in the ZSET (using a set intersection) before issuing the ZADD command. However, this isn't bullet-proof, because it doesn't look like we will be able to do this within the context of a MULTI/EXEC. 

Does anyone have any other ideas?

Coincidentally, this was posted on the exact same day that the WATCH command made it into the Redis master branch.

A solution (in pseudocode, with thanks to Salvatore):

    FUNCTION ZADDNX(key,score,element): 
        WATCH key
        MULTI
        IF (ZSCORE key element) == NIL 
            ZADD key score element
            retval = EXEC 
            IF retval == NIL: 
                ZADDNX(key,score,element)
        ELSE
            DISCARD

(In this example: if the element already exists in the ZSET, then do nothing.)