# Observed CDF of a vector

The empirical distribution function, or observed CDF, comes up a lot in my field of study. Here I have implemented a simple/naive version that performs the calculation, and it works fine on reasonably small sample sizes. But when I start working with ratios derived from river discharge data, now I have a million observations, and my cdf function has started to slow down.

Is there some low-lying fruit to be plucked here to improve throughput? I appreciate your help.

``````fn cdf(x: &[f64]) -> Vec<(f64, f64)> {
let ln = x.len() as f64;
let mut x_ord = x.to_vec();
x_ord.sort_by(|a, b| a.partial_cmp(b).unwrap());
x_ord.dedup();
let mut cdf = Vec::new();
for i in x_ord {
let num = x.par_iter().filter(|x| **x <= i).count() as f64 / ln;
cdf.push((i, num));
}
cdf
}

``````

View in playground

Since you're probably going to be sampling randomly from this using a binary search on the second `f64`, and you factor in duplicates, there's no need to sort or dedup or count, just assign an incremental percent to each one.

So I can skip the initial sort and dedup as shown below, is this what you meant? How do I make count go away?

``````pub fn cdf(x: &[f64]) -> Vec<f64> {
let ln = x.len() as f64;
let mut y = Vec::new();
for i in x {
y.push(x.iter().filter(|x| *x <= i).count() as f64 / ln);
}
y
}
``````

Once you sort the inputs, the count of elements `<=` any unique element is one greater than it's index; for duplicate elements, the same is true of the element at the greatest index. So you could just reduce the sorted `Vec`.

This could maybe be replaced by `dedup_by`, if it has enough guarantees, in which case you could map the input to tuples and sort those to avoid one of the two allocations. Didn't seem worth attempting as a first go though.

1 Like

The code example really helps, I see what you both mean now.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.