Speaking the RESP protocol
Step 2 of 10, and it starts from nothing. Step 1
drew the map and committed no code, so the project begins here: cargo init --name mini-redis
inside the examples repository, or git checkout mini-ch02 once the tag has been pushed. By the end
of this step there is a binary that binds 127.0.0.1:6379, reads one RESP2 frame off the wire, and
answers PING. No crates, no runtime, one file.
The wire format
Section titled “The wire format”RESP2 is a text protocol with a length prefix wherever a length matters. A frame opens with one byte that says what it is, and every line ends with CRLF — two bytes, not one, and the decoder has to agree about both.
Five types carry data, and a sixth value means “nothing”:
| First byte | Frame | Wire form |
|---|---|---|
+ |
simple string | +PONG\r\n |
- |
error | -ERR unknown command 'foo'\r\n |
: |
integer | :100\r\n, signed |
$ |
bulk string | $5\r\nhello\r\n |
* |
array | *2\r\n then two frames |
$-1 / *-1 |
null | a bulk string that is not there |
The length prefix is the whole point. A bulk string’s payload can contain CRLF, or NUL, or invalid UTF-8, and the reader still knows where it ends, because the reader counts bytes instead of scanning for a terminator. Arrays nest to whatever depth the client likes.
Every request is an array of bulk strings. Here is what the socket actually receives when the real
redis-cli 7.2.4 sends PING, captured off a live connection while this step was written:
*1\r\n$4\r\nping\r\nFourteen bytes. *1 is an array of one element, $4 is a bulk string of four bytes, ping is the
payload, and the command name arrives lowercase — redis-cli does not bother with capitals, and real
Redis does not care. The clone has to agree, which is why command matching later in this file folds
case.
Writing a frame
Section titled “Writing a frame”Encoding holds no decisions, so it gets no ceremony: a tag byte, the content, CRLF, recursing into
array elements. Frame::encode appends onto a Vec<u8> the caller owns, which is what lets a reply
be built in place and written with one write_all:
+OK\r\n Frame::Simple("OK"):-2\r\n Frame::Integer(-2)$5\r\nhello\r\n Frame::Bulk(b"hello")$-1\r\n Frame::Null*2\r\n:1\r\n$3\r\ntwo\r\n Frame::Array([Integer(1), Bulk(b"two")])Integers are i64, not u64, because TTL answers -1 and -2 at step 5, and a frame type that
cannot hold the reply is a frame type you get to rewrite later.
Reading a frame
Section titled “Reading a frame”This is the half with the decisions in it. The decoder takes a &[u8], and returns the frame
together with the number of bytes it consumed, so the caller can find the next frame in the same
buffer. Three helpers do the work: read_line scans to the next CRLF and steps past it, take
counts out n bytes, and expect_crlf checks the two bytes that close a bulk string’s payload. Two
more small ones — length, which parses a count line and reads a negative as null, and text,
which refuses a line that is not UTF-8 — keep the main match arms readable.
The distinction that matters is in the error type. Incomplete means the buffer ended inside the
frame: the rest of it may still be on the wire, and nothing is wrong yet. An empty buffer is
Incomplete, not an error. Invalid means every byte is there and the bytes are still not RESP.
A parser that reports one error for both cannot tell “wait” from “no”, and step 3 is entirely about
being able to wait.
Here is the reader half of src/main.rs with a main that decodes the captured bytes above:
// The codec on its own: the reader half of src/main.rs, with a main that// decodes the bytes the real redis-cli sends for PING.
#[derive(Debug, PartialEq)]enum Frame { Simple(String), Error(String), Integer(i64), Bulk(Vec<u8>), Null, Array(Vec<Frame>),}
#[derive(Debug, PartialEq)]enum DecodeError { Incomplete, Invalid(String),}
fn decode(buf: &[u8]) -> Result<(Frame, usize), DecodeError> { let mut pos = 0; let frame = decode_value(buf, &mut pos)?; Ok((frame, pos))}
fn decode_value(buf: &[u8], pos: &mut usize) -> Result<Frame, DecodeError> { let tag = *buf.get(*pos).ok_or(DecodeError::Incomplete)?; *pos += 1; match tag { b'+' => Ok(Frame::Simple(text(read_line(buf, pos)?)?.to_string())), b'-' => Ok(Frame::Error(text(read_line(buf, pos)?)?.to_string())), b':' => { let line = text(read_line(buf, pos)?)?; let n: i64 = line .parse() .map_err(|_| DecodeError::Invalid(format!("bad integer {line:?}")))?; Ok(Frame::Integer(n)) } b'$' => match length(read_line(buf, pos)?)? { None => Ok(Frame::Null), Some(len) => { let data = take(buf, pos, len)?; expect_crlf(buf, pos)?; Ok(Frame::Bulk(data.to_vec())) } }, b'*' => match length(read_line(buf, pos)?)? { None => Ok(Frame::Null), Some(count) => { let mut items = Vec::new(); for _ in 0..count { items.push(decode_value(buf, pos)?); } Ok(Frame::Array(items)) } }, other => Err(DecodeError::Invalid(format!("unknown type byte {other:#04x}"))), }}
fn read_line<'a>(buf: &'a [u8], pos: &mut usize) -> Result<&'a [u8], DecodeError> { let rest = buf.get(*pos..).ok_or(DecodeError::Incomplete)?; match rest.windows(2).position(|pair| pair == b"\r\n") { Some(end) => { let line = &rest[..end]; *pos += end + 2; Ok(line) } None => Err(DecodeError::Incomplete), }}
fn take<'a>(buf: &'a [u8], pos: &mut usize, n: usize) -> Result<&'a [u8], DecodeError> { let end = pos .checked_add(n) .ok_or_else(|| DecodeError::Invalid("length too large".to_string()))?; let slice = buf.get(*pos..end).ok_or(DecodeError::Incomplete)?; *pos = end; Ok(slice)}
fn expect_crlf(buf: &[u8], pos: &mut usize) -> Result<(), DecodeError> { match buf.get(*pos..*pos + 2) { Some([b'\r', b'\n']) => { *pos += 2; Ok(()) } Some(_) => Err(DecodeError::Invalid( "bulk string not followed by CRLF".to_string(), )), None => Err(DecodeError::Incomplete), }}
fn length(line: &[u8]) -> Result<Option<usize>, DecodeError> { let line = text(line)?; let n: i64 = line .parse() .map_err(|_| DecodeError::Invalid(format!("bad length {line:?}")))?; if n < 0 { Ok(None) } else { Ok(Some( usize::try_from(n).map_err(|_| DecodeError::Invalid("length too large".into()))?, )) }}
fn text(bytes: &[u8]) -> Result<&str, DecodeError> { std::str::from_utf8(bytes).map_err(|_| DecodeError::Invalid("not UTF-8".to_string()))}
fn main() { // Captured off a real socket: redis-cli 7.2.4 sends the command name // lowercase. let wire: &[u8] = b"*1\r\n$4\r\nping\r\n"; let (frame, used) = decode(wire).expect("a complete frame"); println!("wire {:?}", std::str::from_utf8(wire).unwrap()); println!("frame {frame:?}"); println!("used {used} of {} bytes", wire.len());
// The same frame with its closing CRLF still in flight. println!("short {:?}", decode(b"*1\r\n$4\r\nping"));}wire "*1\r\n$4\r\nping\r\n"frame Array([Bulk([112, 105, 110, 103])])used 14 of 14 bytesshort Err(Incomplete)Two details worth a second look. Bulk holds Vec<u8>, not String, which is why the debug print
shows [112, 105, 110, 103] instead of "ping": a bulk string is bytes, and the frame layer has
no business deciding it is text. The command layer makes that decision, once, where it reads the
command name. And the array arm builds its Vec with push rather than with_capacity, because
the count is the client’s claim while the buffer is the truth — a header announcing a million
elements costs four bytes and buys nothing. The recursion is bounded the same way: every nesting
level consumes at least four bytes of buffer, so a 4 KiB chunk caps the depth at about a thousand.
The server
Section titled “The server”main binds the port, prints one line, and accepts connections in a loop. Each connection goes to
serve, which does the same three things until the client hangs up: read one chunk, decode one
frame, write one reply. A read that returns zero bytes is the end of the connection, not an error.
execute is the command table, and at this step it is one command deep. Command names fold to
upper case first, because the captured bytes above are lowercase. PING alone answers +PONG;
PING with one argument echoes that argument back as a bulk string, which is what the real server
does — sent ping hello to a real redis-server 7.2.4 and the socket answered $5\r\nhello\r\n.
Two arguments answer the real server’s own wording, captured the same way: -ERR wrong number of arguments for 'ping' command. Anything else is -ERR unknown command '<name>'.
Here is the whole file, byte for byte what the tag mini-ch02 contains:
//! mini-redis, step 2: the RESP2 codec written by hand, and a server that//! answers `PING`.//!//! Start state: nothing — this step creates the project. End state: the tag//! `mini-ch02`, a server that binds 127.0.0.1:6379 and speaks one command.
use std::io::{ErrorKind, Read, Write};use std::net::{TcpListener, TcpStream};
/// One RESP2 value: the six things a frame can be.#[derive(Debug, PartialEq)]enum Frame { /// `+OK\r\n` — a short status line, no length prefix. Simple(String), /// `-ERR ...\r\n` — an error, the same shape as a simple string. Error(String), /// `:42\r\n` — signed, because `TTL` answers -1 and -2 (step 5). Integer(i64), /// `$5\r\nhello\r\n` — length-prefixed bytes. Bytes, not text: a bulk /// string may hold anything, CRLF included. Bulk(Vec<u8>), /// `$-1\r\n` — a bulk string that is not there. Null, /// `*2\r\n` followed by two frames — how every request arrives. Array(Vec<Frame>),}
impl Frame { /// Write the frame's wire form onto `dst`. fn encode(&self, dst: &mut Vec<u8>) { match self { Frame::Simple(s) => { dst.push(b'+'); dst.extend_from_slice(s.as_bytes()); dst.extend_from_slice(b"\r\n"); } Frame::Error(s) => { dst.push(b'-'); dst.extend_from_slice(s.as_bytes()); dst.extend_from_slice(b"\r\n"); } Frame::Integer(n) => { dst.push(b':'); dst.extend_from_slice(n.to_string().as_bytes()); dst.extend_from_slice(b"\r\n"); } Frame::Bulk(b) => { dst.push(b'$'); dst.extend_from_slice(b.len().to_string().as_bytes()); dst.extend_from_slice(b"\r\n"); dst.extend_from_slice(b); dst.extend_from_slice(b"\r\n"); } Frame::Null => dst.extend_from_slice(b"$-1\r\n"), Frame::Array(items) => { dst.push(b'*'); dst.extend_from_slice(items.len().to_string().as_bytes()); dst.extend_from_slice(b"\r\n"); for item in items { item.encode(dst); } } } }}
/// Why a buffer did not hold one whole frame.#[derive(Debug, PartialEq)]enum DecodeError { /// The buffer ends inside the frame. Not a protocol error: the rest of /// the frame may still be on the wire. Step 3 is this variant's whole /// reason for existing. Incomplete, /// The bytes are all there, and they are not RESP. Invalid(String),}
/// Decode one complete frame from the front of `buf`, and report how many/// bytes it used. Anything after that belongs to the next frame.fn decode(buf: &[u8]) -> Result<(Frame, usize), DecodeError> { let mut pos = 0; let frame = decode_value(buf, &mut pos)?; Ok((frame, pos))}
fn decode_value(buf: &[u8], pos: &mut usize) -> Result<Frame, DecodeError> { let tag = *buf.get(*pos).ok_or(DecodeError::Incomplete)?; *pos += 1; match tag { b'+' => Ok(Frame::Simple(text(read_line(buf, pos)?)?.to_string())), b'-' => Ok(Frame::Error(text(read_line(buf, pos)?)?.to_string())), b':' => { let line = text(read_line(buf, pos)?)?; let n: i64 = line .parse() .map_err(|_| DecodeError::Invalid(format!("bad integer {line:?}")))?; Ok(Frame::Integer(n)) } b'$' => match length(read_line(buf, pos)?)? { None => Ok(Frame::Null), Some(len) => { let data = take(buf, pos, len)?; expect_crlf(buf, pos)?; Ok(Frame::Bulk(data.to_vec())) } }, b'*' => match length(read_line(buf, pos)?)? { None => Ok(Frame::Null), Some(count) => { // No with_capacity: the count is the client's claim, the // buffer is the truth, and every element costs bytes. let mut items = Vec::new(); for _ in 0..count { items.push(decode_value(buf, pos)?); } Ok(Frame::Array(items)) } }, other => Err(DecodeError::Invalid(format!("unknown type byte {other:#04x}"))), }}
/// The bytes before the next CRLF, with `pos` moved past it.fn read_line<'a>(buf: &'a [u8], pos: &mut usize) -> Result<&'a [u8], DecodeError> { let rest = buf.get(*pos..).ok_or(DecodeError::Incomplete)?; match rest.windows(2).position(|pair| pair == b"\r\n") { Some(end) => { let line = &rest[..end]; *pos += end + 2; Ok(line) } None => Err(DecodeError::Incomplete), }}
/// `n` bytes, or Incomplete when the buffer is short of them.fn take<'a>(buf: &'a [u8], pos: &mut usize, n: usize) -> Result<&'a [u8], DecodeError> { let end = pos .checked_add(n) .ok_or_else(|| DecodeError::Invalid("length too large".to_string()))?; let slice = buf.get(*pos..end).ok_or(DecodeError::Incomplete)?; *pos = end; Ok(slice)}
/// The CRLF that closes a bulk string's payload.fn expect_crlf(buf: &[u8], pos: &mut usize) -> Result<(), DecodeError> { match buf.get(*pos..*pos + 2) { Some([b'\r', b'\n']) => { *pos += 2; Ok(()) } Some(_) => Err(DecodeError::Invalid( "bulk string not followed by CRLF".to_string(), )), None => Err(DecodeError::Incomplete), }}
/// A length or a count line. Negative means "null"; anything else has to/// parse and fit in a usize.fn length(line: &[u8]) -> Result<Option<usize>, DecodeError> { let line = text(line)?; let n: i64 = line .parse() .map_err(|_| DecodeError::Invalid(format!("bad length {line:?}")))?; if n < 0 { Ok(None) } else { Ok(Some( usize::try_from(n).map_err(|_| DecodeError::Invalid("length too large".into()))?, )) }}
/// Simple strings and lengths are ASCII in practice; reject the rest rather/// than guess.fn text(bytes: &[u8]) -> Result<&str, DecodeError> { std::str::from_utf8(bytes).map_err(|_| DecodeError::Invalid("not UTF-8".to_string()))}
fn main() -> std::io::Result<()> { let listener = TcpListener::bind("127.0.0.1:6379")?; println!("listening on 127.0.0.1:6379"); for stream in listener.incoming() { let mut stream = stream?; if let Err(err) = serve(&mut stream) { eprintln!("connection error: {err}"); } } Ok(())}
/// One connection: read, decode, answer, repeat. The read is a single/// `read` call over one chunk, so a frame split across two packets ends the/// connection, and a second frame in the same chunk is dropped. Both gaps/// are the same missing buffer, and step 3 is that buffer.fn serve(stream: &mut TcpStream) -> std::io::Result<()> { let mut chunk = [0u8; 4096]; loop { let n = stream.read(&mut chunk)?; if n == 0 { return Ok(()); // the client hung up } // `used` is ignored on purpose: whatever follows the frame waits in // a buffer that does not exist yet. let (frame, _used) = decode(&chunk[..n]).map_err(|err| match err { DecodeError::Incomplete => { std::io::Error::new(ErrorKind::UnexpectedEof, "a frame arrived in pieces") } DecodeError::Invalid(why) => std::io::Error::new(ErrorKind::InvalidData, why), })?; let mut reply = Vec::new(); execute(&frame).encode(&mut reply); stream.write_all(&reply)?; }}
/// The command table, which is one command deep. Names are/// case-insensitive: the redis-cli on this machine sends `ping` lowercase,/// captured off the wire while this step was written.fn execute(frame: &Frame) -> Frame { let parts = match frame { Frame::Array(parts) => parts, _ => return Frame::Error("ERR expected an array of bulk strings".to_string()), }; let name: Vec<u8> = match parts.first() { Some(Frame::Bulk(name)) => name.to_ascii_uppercase(), _ => return Frame::Error("ERR expected a command name".to_string()), }; if name == b"PING" { match parts.len() { 1 => Frame::Simple("PONG".to_string()), 2 => match &parts[1] { Frame::Bulk(message) => Frame::Bulk(message.clone()), _ => wrong_args("ping"), }, _ => wrong_args("ping"), } } else { Frame::Error(format!( "ERR unknown command '{}'", String::from_utf8_lossy(&name).to_ascii_lowercase() )) }}
fn wrong_args(command: &str) -> Frame { Frame::Error(format!( "ERR wrong number of arguments for '{command}' command" ))}
#[cfg(test)]mod tests { use super::*; use std::net::Shutdown;
fn bytes(frame: &Frame) -> Vec<u8> { let mut out = Vec::new(); frame.encode(&mut out); out }
#[test] fn every_frame_type_round_trips() { let frames = [ Frame::Simple("OK".to_string()), Frame::Error("ERR unknown command 'foo'".to_string()), Frame::Integer(-2), Frame::Bulk(b"hello\r\nworld".to_vec()), Frame::Bulk(vec![0x00, 0xff]), Frame::Null, Frame::Array(vec![Frame::Integer(1), Frame::Bulk(b"two".to_vec())]), ]; for frame in &frames { let wire = bytes(frame); let (decoded, used) = decode(&wire).expect("a whole frame decodes"); assert_eq!(&decoded, frame); assert_eq!(used, wire.len(), "the frame used the whole buffer"); } }
#[test] fn the_exact_bytes_the_real_redis_cli_sends_for_ping() { // Captured off a real socket: redis-cli 7.2.4 sends the command // name lowercase. let wire = b"*1\r\n$4\r\nping\r\n"; let (frame, used) = decode(wire).expect("decodes"); assert_eq!(frame, Frame::Array(vec![Frame::Bulk(b"ping".to_vec())])); assert_eq!(used, wire.len()); assert_eq!(execute(&frame), Frame::Simple("PONG".to_string())); }
#[test] fn a_frame_in_two_pieces_is_incomplete_not_invalid() { assert_eq!(decode(b""), Err(DecodeError::Incomplete)); assert_eq!(decode(b"*1\r\n$4\r\npi"), Err(DecodeError::Incomplete)); // The payload is all there; the closing CRLF is still in flight. assert_eq!(decode(b"*1\r\n$4\r\nping"), Err(DecodeError::Incomplete)); }
#[test] fn bytes_that_are_not_resp_are_invalid() { assert!(matches!(decode(b"?\r\n"), Err(DecodeError::Invalid(_)))); assert!(matches!(decode(b":soon\r\n"), Err(DecodeError::Invalid(_)))); assert!(matches!(decode(b"*x\r\n"), Err(DecodeError::Invalid(_)))); assert!(matches!(decode(b"$4\r\nhiXYab"), Err(DecodeError::Invalid(_)))); }
#[test] fn one_decode_consumes_exactly_one_frame() { let wire = b"*1\r\n$4\r\nping\r\n*1\r\n$4\r\nping\r\n"; let (frame, used) = decode(wire).expect("decodes"); assert_eq!(used, 14); assert_eq!(execute(&frame), Frame::Simple("PONG".to_string())); }
#[test] fn ping_with_an_argument_echoes_it() { let wire = b"*2\r\n$4\r\nping\r\n$5\r\nhello\r\n"; let (frame, _) = decode(wire).expect("decodes"); assert_eq!(execute(&frame), Frame::Bulk(b"hello".to_vec())); }
#[test] fn ping_over_a_real_socket() { let listener = TcpListener::bind("127.0.0.1:0").expect("binds an ephemeral port"); let addr = listener.local_addr().expect("has an address"); let server = std::thread::spawn(move || { let (mut stream, _) = listener.accept().expect("accepts one connection"); serve(&mut stream).expect("serves until the client hangs up"); }); let mut client = TcpStream::connect(addr).expect("connects"); client .write_all(b"*1\r\n$4\r\nping\r\n") .expect("writes a frame"); client.shutdown(Shutdown::Write).expect("half-closes"); let mut reply = Vec::new(); client.read_to_end(&mut reply).expect("reads the reply"); server.join().expect("the server thread did not panic"); assert_eq!(reply, b"+PONG\r\n"); }}Seven tests pin the behaviour down. Every frame type round-trips through encode and decode. The
exact captured bytes decode to the expected frame. A frame cut in half — including the case where
the payload has arrived and the closing CRLF has not — is Incomplete, while a wrong type byte, a
non-numeric length, and a payload followed by something other than CRLF are all Invalid. One
decode consumes exactly one frame and reports where the next one starts. And one test speaks over a
real socket: an ephemeral listener, a server thread, a client that writes *1\r\n$4\r\nping\r\n
and half-closes, and an assertion that the bytes coming back are +PONG\r\n.
$ cargo test Compiling mini-redis v0.1.0 Finished `test` profile [unoptimized + debuginfo] target(s) in 0.33s Running unittests src/main.rs (/Volumes/RustBuild/tracks/debug/deps/mini_redis-f7b0b7e7128a87e0)
running 7 teststest tests::bytes_that_are_not_resp_are_invalid ... oktest tests::a_frame_in_two_pieces_is_incomplete_not_invalid ... oktest tests::every_frame_type_round_trips ... oktest tests::ping_with_an_argument_echoes_it ... oktest tests::one_decode_consumes_exactly_one_frame ... oktest tests::the_exact_bytes_the_real_redis_cli_sends_for_ping ... oktest tests::ping_over_a_real_socket ... ok
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00sThe order the tests print in varies between runs; the set does not. The Running line names this
machine’s shared build cache; elsewhere it reads target/debug/deps/mini_redis-….
Seeing it run
Section titled “Seeing it run”cargo run on this machine stops before it listens, because a real redis-server already owns
127.0.0.1:6379:
$ cargo run Running `/Volumes/RustBuild/tracks/debug/mini-redis`Error: Os { code: 48, kind: AddrInUse, message: "Address already in use" }On a machine where the port is free the server prints listening on 127.0.0.1:6379 and waits. To
check the clone against the real client without disturbing the server that owns the port, the same
binary was bound to a scratch port and the real redis-cli pointed at it:
$ redis-cli -p 7777 pingPONG$ redis-cli -p 7777 ping hellohello$ redis-cli -p 7777 fooERR unknown command 'foo'$ redis-cli -p 7777 ping a bERR wrong number of arguments for 'ping' commandWhat is deliberately not here
Section titled “What is deliberately not here”- A frame split across two packets ends the connection.
servedoes onereadper frame, so a frame in two pieces decodes asIncompleteand the connection closes witha frame arrived in pieces. Verified by sending*1\r\n$4\r\npi, pausing, thenng\r\n: no reply, and that line in the server’s log. - A second frame in the same chunk is dropped.
decodereports how many bytes it used andserveignores the number. Pipelining — two commands written back to back — loses the second one. Both gaps are the same missing buffer, and step 3 is that buffer. - Inline commands are not spoken. A bare
PING\r\nwith no array around it is legal RESP2, and the real server answers it (captured:+PONG). The clone’s parser accepts arrays only, which is the dialect every client library uses. - A null bulk inside a request array decodes here. The real server refuses one with
-ERR Protocol error: invalid bulk length(captured). Nothing sends that frame yet;GETwill send$-1as a reply at step 4, which is the direction that matters.
Step 3 keeps the bytes that serve throws away: a connection with a buffer, a decoder that
survives a packet boundary, and ECHO to prove it. The track overview
carries the full list as articles land.
Related
Section titled “Related”- Handling errors —
DecodeErroris the shape every fallible parser takes: an enum, aDisplay, and?. - How to convert a Vec of bytes to a String — what the command layer does to a bulk payload when it needs text.
- What we’re building: a Redis clone — step 1, the map this step starts from.