Benchmarks, and what real Redis does

Step 10 of 10. Step 9 ended at the tag mini-ch09 with an integration harness that drives the server over a real socket. This step starts from that tag and ends at mini-ch10, the last one. It adds no commands and fixes no bugs: it points a small load generator at the server, reads the numbers, and then takes stock of what the clone does and does not do next to the real thing.

The new code is one file, src/bin/bench.rs, in the examples repository under mini-redis/ — a second binary that cargo builds alongside the server. Run the server in one terminal and the bench in another:

cargo run --release -- 7790            # terminal 1: the server
cargo run --release --bin bench -- 127.0.0.1:7790 4 10000 16   # terminal 2

What the tests never measured

The twelve integration tests from step 9 prove the server correct. None of them asks how fast it is, and the gap between those two questions is bigger than it looks. Every SET this server has ever answered cost one request and one reply; the store work itself — a hash insert — is nanoseconds. Almost all of the time in a command lives on the path between the two ends of the socket. A benchmark that claims to measure “the server” is really measuring that path, and the first job of a benchmark is to be honest about it.

Two costs hide inside one round trip: the syscall-and-wait cost of sending a request and waiting for its reply, and the actual work the server does. They can be pulled apart with a stand-in server that does no work at all — it answers +OK to whatever arrives. Same client code, same socket, no store. Whatever remains is the per-command cost of talking:

[dependencies]
tokio = { version = "1", features = ["macros", "rt-multi-thread", "net", "io-util"] }

use std::time::Instant;

use tokio::io::{AsyncBufReadExt, AsyncReadExt, AsyncWriteExt, BufReader};
use tokio::net::{TcpListener, TcpStream};

// A stand-in server: one `+OK` per complete request. Every request
// ends in `\n` and contains none inside, so counting newlines per
// read counts whole requests, even when one lands split across reads.
async fn echo_server(listener: TcpListener) {
    loop {
        let (mut sock, _) = listener.accept().await.unwrap();
        tokio::spawn(async move {
            let mut buf = [0u8; 4096];
            loop {
                match sock.read(&mut buf).await {
                    Ok(0) | Err(_) => return,
                    Ok(n) => {
                        let replies = b"+OK\r\n"
                            .repeat(buf[..n].iter().filter(|&&b| b == b'\n').count());
                        if sock.write_all(&replies).await.is_err() {
                            return;
                        }
                    }
                }
            }
        });
    }
}

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let listener = TcpListener::bind("127.0.0.1:0").await?;
    let addr = listener.local_addr()?;
    tokio::spawn(echo_server(listener));

    let mut client = BufReader::new(TcpStream::connect(addr).await?);
    let ping: &[u8] = b"*1\r\n$4\r\nPING\r\n";
    let n = 2_000;

    // sequential: one request out, wait for the reply, repeat
    let start = Instant::now();
    for _ in 0..n {
        client.get_mut().write_all(ping).await?;
        let mut line = Vec::new();
        client.read_until(b'\n', &mut line).await?;
    }
    let plain = n as f64 / start.elapsed().as_secs_f64();

    // pipelined: 16 requests in one write, then 16 replies
    let start = Instant::now();
    let mut left = n;
    while left > 0 {
        let batch = 16.min(left);
        client.get_mut().write_all(&ping.repeat(batch)).await?;
        for _ in 0..batch {
            let mut line = Vec::new();
            client.read_until(b'\n', &mut line).await?;
        }
        left -= batch;
    }
    let piped = n as f64 / start.elapsed().as_secs_f64();

    println!("sequential : {plain:>8.0} ops/s");
    println!("pipelined  : {piped:>8.0} ops/s");
    Ok(())
}

▶Run this in the Rust Playground

sequential :   288034 ops/s
pipelined  :  1113793 ops/s

On this run the stand-in answers about 288,000 sequential ops per second, and about 1.1 million once the client pipelines — nearly four times as much from the same client, the same socket, and a server that does no work at all. Pipelining does not make the server faster; it buys back the waiting between request and reply, and on an idle loopback most of a round trip is exactly that. Keep the two numbers in mind — they are the levers the real bench pulls below.

A load generator, not a benchmark suite

Benchmark frameworks like criterion answer a different question: is this function faster than it was. What the server needs is closer to a load generator — many connections, many commands, wall clock around the whole batch — and that is a hundred lines of the same protocol code the tests already use, pointed at the server in a loop instead of once.

It lives in src/bin/bench.rs. Cargo builds every file in src/bin/ as its own binary, so cargo run --bin bench works with no Cargo.toml changes. The file cannot import the server’s code — binaries do not share code with each other, and tests/client.rs is a separate crate on purpose — so the small protocol pieces are repeated here. That is the shape step 9 promised: the same encoder, the same read-one-reply loop, a different thing to do with it.

The encoding side is the array form of RESP, the same bytes a real client would put on the wire:

src/bin/bench.rs — the request, and reading one reply

/// One encoded command, ready to write.
fn encode(cmd: &[&[u8]]) -> Vec<u8> {
    let mut out = format!("*{}\r\n", cmd.len()).into_bytes();
    for arg in cmd {
        out.extend_from_slice(format!("${}\r\n", arg.len()).as_bytes());
        out.extend_from_slice(arg);
        out.extend_from_slice(b"\r\n");
    }
    out
}

/// Read exactly one reply, whatever its length. The first byte says
/// which kind it is; only bulk strings need a second read.
async fn read_reply(r: &mut BufReader<TcpStream>) -> std::io::Result<()> {
    let mut line = Vec::new();
    r.read_until(b'\n', &mut line).await?;
    match line[0] {
        b'+' | b'-' | b':' | b',' | b'_' | b'%' | b'>' | b'~' => Ok(()),
        b'$' => {
            let n: i64 = std::str::from_utf8(&line[1..line.len() - 2])
                .unwrap_or("")
                .trim()
                .parse()
                .unwrap_or(0);
            if n > 0 {
                let mut rest = vec![0u8; n as usize + 2]; // payload + CRLF
                r.read_exact(&mut rest).await?;
            }
            Ok(())
        }
        _ => Err(std::io::Error::new(
            std::io::ErrorKind::InvalidData,
            "not a reply",
        )),
    }
}

The interesting function is the worker. One connection, ops commands, and a pipeline depth: with pipe = 1 it is a strict request-reply loop; with pipe = 16 it writes sixteen requests and only then reads sixteen replies. Nothing about the server changes — step 3 built a decoder that pulls frames off a buffer instead of assuming one request per read, and that decision is what makes pipelining work at all:

src/bin/bench.rs — one connection's worth of work

/// One connection's worth of work: `ops` commands, `pipe` at a time.
async fn worker(addr: String, mode: Mode, id: usize, ops: usize, pipe: usize) -> std::io::Result<()> {
    let mut r = BufReader::new(TcpStream::connect(&addr).await?);
    let key = format!("k{id}");
    let value = vec![b'x'; 64];
    let req = match mode {
        Mode::Set => encode(&[b"SET", key.as_bytes(), &value]),
        Mode::Get => encode(&[b"GET", key.as_bytes()]),
    };
    if pipe <= 1 {
        for _ in 0..ops {
            r.get_mut().write_all(&req).await?;
            read_reply(&mut r).await?;
        }
    } else {
        let mut done = 0;
        while done < ops {
            let batch = pipe.min(ops - done);
            for _ in 0..batch {
                r.get_mut().write_all(&req).await?;
            }
            for _ in 0..batch {
                read_reply(&mut r).await?;
            }
            done += batch;
        }
    }
    Ok(())
}

A phase is one clock around many connections: run_phase spawns one worker per connection and waits for every one of them, so each line below is one wall-clock time for a whole batch, not an average of per-worker times. report turns a phase into a line, and Mode is just which command the workers send:

src/bin/bench.rs — a phase, and its report line

#[derive(Clone, Copy)]
enum Mode {
    Set,
    Get,
}

async fn run_phase(addr: &str, mode: Mode, conns: usize, ops: usize, pipe: usize) -> std::io::Result<Duration> {
    let start = Instant::now();
    let mut handles = Vec::new();
    for id in 0..conns {
        handles.push(tokio::spawn(worker(addr.to_string(), mode, id, ops, pipe)));
    }
    for h in handles {
        h.await.unwrap()?;
    }
    Ok(start.elapsed())
}

fn report(name: &str, ops: usize, t: Duration, pipe: usize) {
    let secs = t.as_secs_f64();
    println!(
        "{name:<5} pipeline={pipe:<3} {ops:>7} ops in {secs:7.3}s  {:>10.0} ops/s",
        ops as f64 / secs
    );
}

main runs four phases — SET and GET, sequential and pipelined. Before timing anything it sends one PING — a server that is not running should produce one clear error naming the address, not a run that fails halfway through with connection refused from worker 3 of 4:

src/bin/bench.rs — the phases, and the probe

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let mut args = std::env::args().skip(1);
    let addr = args.next().unwrap_or_else(|| "127.0.0.1:6379".to_string());
    let conns: usize = args.next().map(|v| v.parse().expect("connections")).unwrap_or(4);
    let ops: usize = args.next().map(|v| v.parse().expect("ops per connection")).unwrap_or(10_000);
    let pipe: usize = args.next().map(|v| v.parse().expect("pipeline depth")).unwrap_or(16);

    // One PING first: if the server is not up, say so now, with the
    // address in the message, instead of failing halfway through a run.
    let mut probe = BufReader::new(TcpStream::connect(&addr).await.map_err(|e| {
        std::io::Error::new(e.kind(), format!("cannot reach {addr}: {e}"))
    })?);
    probe.get_mut().write_all(b"*1\r\n$4\r\nPING\r\n").await?;
    let mut pong = Vec::new();
    probe.read_until(b'\n', &mut pong).await?;
    println!("ping -> {}", String::from_utf8_lossy(&pong).trim_end());

    println!("{addr}  connections={conns}  ops-per-connection={ops}");
    let total = conns * ops;
    let t = run_phase(&addr, Mode::Set, conns, ops, 1).await?;
    report("SET", total, t, 1);
    let t = run_phase(&addr, Mode::Get, conns, ops, 1).await?;
    report("GET", total, t, 1);
    let t = run_phase(&addr, Mode::Set, conns, ops, pipe).await?;
    report("SET", total, t, pipe);
    let t = run_phase(&addr, Mode::Get, conns, ops, pipe).await?;
    report("GET", total, t, pipe);
    Ok(())
}

The numbers

One connection first, then four. Everything here is a real run of the binary built from this step’s tag, on one machine, on localhost; another machine moves the absolute numbers, and rerunning on the same machine moves them a little too. The ratios are the part worth reading:

$ cargo run --release --bin bench -- 127.0.0.1:7790 1 10000 16
ping -> +PONG
127.0.0.1:7790  connections=1  ops-per-connection=10000
SET   pipeline=1     10000 ops in   0.200s       49876 ops/s
GET   pipeline=1     10000 ops in   0.198s       50459 ops/s
SET   pipeline=16    10000 ops in   0.038s      260338 ops/s
GET   pipeline=16    10000 ops in   0.040s      251855 ops/s

One connection, one request in flight: about 50,000 ops per second, which is about twenty microseconds per round trip, store work included. The stand-in’s no-work round trip was about 3.5 microseconds; the rest is the server actually working — reading frames off the buffer, taking the store lock, waking the right task. The shape is the one the stand-in predicted: the wait between request and reply dominates, and the hash insert is rounding error.

Pipelining sixteen requests per write multiplies the same single connection by five. The requests still cost the same to execute; what changes is that sixteen of them share one syscall and one wait instead of paying for their own each. This is why real Redis clients pipeline, and why Redis itself treats MULTI/EXEC as a pipeline: amortising the round trip is the cheapest performance win a protocol can offer.

$ cargo run --release --bin bench -- 127.0.0.1:7790 4 10000 16
ping -> +PONG
127.0.0.1:7790  connections=4  ops-per-connection=10000
SET   pipeline=1     40000 ops in   0.394s      101475 ops/s
GET   pipeline=1     40000 ops in   0.363s      110069 ops/s
SET   pipeline=16    40000 ops in   0.095s      422606 ops/s
GET   pipeline=16    40000 ops in   0.105s      380257 ops/s

Four connections do not buy four connections’ worth. Sequential throughput doubles, not quadruples: 100k rather than the 200k that four 50k connections would suggest. Something in the server serialises part of the work, and the obvious suspect is the store lock that every command takes in step 4 — but this bench cannot separate time spent waiting for the lock from time spent on the socket, so that stays a hypothesis, not a finding. Building the tool that separates them is exactly the kind of step 11 this project would earn if it kept going.

None of these numbers are comparable with real Redis’s published ones. Real Redis is benchmarked with redis-benchmark, on tuned hardware, with its own pipelining defaults. What this bench is for is narrower and more useful: point it at the tree before and after a change, and the ratios say whether the change was worth making.

What real Redis does differently

The gaps left along the way were mostly framed as “too much for one step”. Seen from the top, they fall into three groups.

Execution model. Real Redis runs command execution on one thread — since Redis 6.0, I/O threads read and write sockets, but commands themselves execute serially. There is no store lock because there is nothing to lock; MULTI is trivially atomic because nothing can interleave with it. Step 4 chose one Tokio task per connection with a shared Mutex<HashMap>, which scales across cores but pays for every command with a lock, and makes step 8’s transaction queue genuinely necessary. The doubling-not-quadrupling above is that trade showing up in a number.

Durability and memory. Real Redis appends every write to an AOF with fsync policies (everysec being the usual choice) and snapshots RDB in a forked child; step 6 chose a SAVE snapshot only, which is honest but loses everything since the last save, and blocks while it writes. Real Redis also enforces maxmemory and evicts by LRU or LFU; this store grows until the process dies. Expiry is closest of the three: step 5’s sweeper is a slower version of Redis’s active-expiry cycle, which samples keys with deadlines instead of sweeping everything.

The rest of the surface. No lists, hashes, or streams — the command set is the twelve from step 1. No WATCH, so no check-and-set retry loops. No replication, no cluster, RESP2 only. None of those are performance gaps; they are product gaps, and each one is a well-shaped next project: WATCH needs a per-key watcher registry, an append-only file needs a replay on startup next to the snapshot loader from step 6, and replication needs most of the client code from step 9 pointed from one server at another.

Where the project ends

The last tag:

git checkout mini-ch10

At that tag the project is a server that speaks RESP, holds values with deadlines, survives a restart with its snapshot, fans pub/sub messages out, queues transactions per connection, and carries the proof of all of it: unit tests, a black-box integration harness, and a bench that puts a number on the whole thing. That is a real project, and the honest version of “build a Redis clone in Rust” — small enough to hold in your head, complete enough that every piece was actually made to work.

The gaps above are the syllabus for a second edition, if one is ever wanted. Until then, the track overview lists all ten steps in order.

Related

• Testing a network server — step 9, whose client shape the bench repeats; the probe and the worker are that test client doing a different job.
• Handling concurrent connections — step 4, the task-per-connection-plus-lock design whose cost the four-connection numbers show.
• Persistence with a snapshot — step 6, the SAVE path the bench deliberately leaves out, and the biggest durability gap next to real Redis.