How to download a file over HTTP

Short answer: Call response.chunk().await in a loop and write each chunk to a tokio::fs::File. The body travels from the socket to the disk a piece at a time, so a 4 GB file never sits in memory. response.bytes().await is the one-call version, and it collects the whole body into memory before a byte reaches the file.

The dependencies

The same two crates as How to make a GET request, with two tokio features named: fs for tokio::fs::File, and io-util for the AsyncWriteExt trait that write_all comes from. Neither is in tokio’s default set.

[dependencies]
reqwest = "0.13.5"
tokio = { version = "1.53.3", features = ["macros", "rt-multi-thread", "fs", "io-util"] }

cargo new download gives you the project; these two lines go under the [dependencies] it already wrote. reqwest’s default features include TLS, so the same code works against an https:// URL unchanged.

The test server

Every block below runs against the same small server, written with nothing but std. It answers any request with a fixed 49,152-byte body and a Content-Length header. The OS picks the port, so the URL is different on every run, which is why the program prints the byte count and not the URL.

Stream the body to a file

use std::error::Error;
use std::io::{BufRead, BufReader, Write};
use std::net::TcpListener;
use std::thread;
use tokio::fs::File;
use tokio::io::AsyncWriteExt;

/// A one-file test server. Every request is answered with the same fixed body.
fn spawn_server() -> String {
    let listener = TcpListener::bind("127.0.0.1:0").unwrap();
    let base = format!("http://{}", listener.local_addr().unwrap());
    thread::spawn(move || {
        for stream in listener.incoming() {
            let mut stream = stream.unwrap();
            let mut reader = BufReader::new(stream.try_clone().unwrap());
            let mut request_line = String::new();
            reader.read_line(&mut request_line).unwrap();
            let body: String = (1..=4096).map(|n| format!("record {n:04}\n")).collect();
            write!(
                stream,
                "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
                body.len()
            )
            .unwrap();
            stream.flush().unwrap();
        }
    });
    base
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let base = spawn_server();
    let url = format!("{base}/download");

    let mut response = reqwest::get(&url).await?;
    let mut file = File::create("report.txt").await?;
    let mut total = 0;
    while let Some(chunk) = response.chunk().await? {
        file.write_all(&chunk).await?;
        total += chunk.len();
    }
    file.flush().await?;
    println!("wrote {total} bytes to report.txt");
    Ok(())
}

Run this in the Rust Playground

Run it with cargo run (cargo’s own Compiling and Finished lines are not shown here):

wrote 49152 bytes to report.txt

chunk() reads one piece of the body and returns Ok(None) once the body is exhausted; that None is what ends the loop. Each chunk goes straight to the file handle, which stays open for the whole download, and total counts what actually crossed the socket. write_all is awaited per chunk, so the file grows as the response arrives rather than all at once at the end.

Check that the file on disk holds what the program counted:

wc -c report.txt

   49152 report.txt

The version that buffers the whole file

use std::error::Error;
use std::io::{BufRead, BufReader, Write};
use std::net::TcpListener;
use std::thread;

/// A one-file test server. Every request is answered with the same fixed body.
fn spawn_server() -> String {
    let listener = TcpListener::bind("127.0.0.1:0").unwrap();
    let base = format!("http://{}", listener.local_addr().unwrap());
    thread::spawn(move || {
        for stream in listener.incoming() {
            let mut stream = stream.unwrap();
            let mut reader = BufReader::new(stream.try_clone().unwrap());
            let mut request_line = String::new();
            reader.read_line(&mut request_line).unwrap();
            let body: String = (1..=4096).map(|n| format!("record {n:04}\n")).collect();
            write!(
                stream,
                "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
                body.len()
            )
            .unwrap();
            stream.flush().unwrap();
        }
    });
    base
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let base = spawn_server();

    let response = reqwest::get(format!("{base}/download")).await?;
    let bytes = response.bytes().await?;
    tokio::fs::write("report.txt", &bytes).await?;
    println!("wrote {} bytes to report.txt", bytes.len());
    Ok(())
}

Run this in the Rust Playground

wrote 49152 bytes to report.txt

Same output, same file, and that is the trap. .bytes() writes nothing until the last byte has arrived, because its job is to hand back the complete body as one Bytes. The whole file is in memory at that point, which is harmless for a 49 KB test file and fatal for a 4 GB film. The streaming loop’s peak is one chunk, whatever the size of the file.

When the download dies halfway

A connection that drops mid-body leaves a truncated report.txt on disk, and nothing in the code above notices. Download to a temporary name and rename on success instead: temp file plus rename, so a crash cannot leave half a file behind, is How to write a file atomically. When the server sends a Content-Length header, response.content_length() reports the size before the first chunk arrives, which is the number a progress indicator needs.

How to parse JSON from an API — the previous article, for when the body is JSON and belongs in a struct.
How to make a GET request — the request itself, and .text() for a body that does fit in memory.
How to write a file atomically — the rename that makes a failed download harmless.

The next article in this chapter takes the URL apart, piece by piece.