Best Way to Process a Large CSV Time-Series Dataset in Rust?

help
1

I’ve been experimenting with Rust for processing larger time-series datasets and I’m trying to figure out the most sensible approach once the files stop being small.

The data itself is pretty simple:



timestamp,symbol,price,volume
1720000000,BTCUSDT,61250.4,0.82
1720000060,BTCUSDT,61271.1,1.14

Some of the historical market data I’m working with comes from different sources, including exports from BYDFi. After cleaning it, the files can easily grow to several million rows.

My first thought was to use the csv crate and process each record one by one instead of loading the whole dataset into memory:



use csv::Reader;
use std::error::Error;

fn main() -> Result<(), Box<dyn Error>> {
    let mut reader = Reader::from_path("market_data.csv")?;

    for result in reader.records() {
        let record = result?;
        // process each row
    }

    Ok(())
}

That seems straightforward when I only need to calculate something simple.

Where I’m getting a little stuck is rolling calculations. For example, if I want to calculate returns, a moving average, or volatility over the previous 100 or 1,000 observations, I obviously need to keep some previous values around.

Would a VecDeque with a fixed maximum size be a reasonable approach here?

I’ve also looked at Polars, which seems much more convenient for this type of analysis, but I’m wondering whether using a dataframe library defeats some of the reason for doing this in Rust in the first place.

For datasets with a few million rows, would you generally:

  1. stream the CSV and maintain a rolling buffer;
  2. load the data with Polars and use its built-in operations; or
  3. use another approach entirely?

I’m coming from more of a Python/data-analysis background, so I’m interested in what would be considered the more idiomatic Rust approach.

Thanks!

4

a circular buffer seems like a perfectly fine approach for analysis on a sliding window of data.

for some cases if you are doing and operation that does not accumulate errors like a sum of integers it's also possible to do something like this

slidingsum[0]=sum(data[0..window_size])
slidingsum[i]=slidingsum[i-1]-data[i-1]+data[i+window_size-1]

this can cover quite a few cases since a sliding average for example is also essentially a sliding sum divided by a constant

5

What does that mean? 10 million? Or 100 million? On average? In the worst-case scenario?

The second question: What is the runtime environment like?

Are you targeting a small single-board computer with 1 or 2 GB of RAM, or a desktop PC or server where your program can use 8 GB or 100 GB of RAM?

You need to be very specific and concrete when making these two design decisions, as they significantly influence your options available to shape you program.

6

I honestly would simply use Polars and call it a day.

7

First thing I'd do is write a program that will spawn a thread per cpu and each will write out raw binary data to files. Then loading and seeking would be easy, without parsing, deserializating, converting bases etc. You can join them together afterwards.

After this you could think of what you wanna do with it later, and any experimentation, and oh boy you'll run the program more then 1 time, will be a lot, like 100x or more times faster, if you have bunary file(s).

And you should read into a fixed size buffer, continuously overwriting it, to keep reusing cache and avoid page faults.

8

Huge files are better be mmaped into virtual memory, especially if you read them sequentally.

If you are going to do lots of analysis, you could first convert csv into something more computer-friendly (e.g. records with fixed row length, represent numbers as integers and all these constants also, to make sure N records fit exactly in one page (4K))

Say, you have 4 fields, each field is 8 bytes, not a big deal to fit into memory.