Direct access compared to interpretive loop for RustDb

I just added a third benchmark test, which accesses the data directly compared to using the interpreted language: bench.rs - source

The test specification is " Each test should first create a table with two columns, insert 8,192 identical rows 'Alice', 1000. Then (the timed part) it should total the second column ( result 8,192,000 ) and do this 1,000 times."

Results running on my laptop:

rustdb direct test took 303 milli-seconds
rustdb test took 488 milli-seconds
sqllite test took 569 milli-seconds

I am somewhat surprised the direct access is significantly faster, as I expected the overhead of iterating through the records to dominate, but anyway it is interesting.

Any suggestions for other databases I could add to the benchmark?

You should use Instant instead of SystemTime if you want to know how long something takes. It's probably not affecting this particular benchmark, but it's important to know.

1 Like

Yes. What I have noticed is a surprising amount of variability in the timings.

For example a test could take 316 milli-seconds, I run it again and it is 183 milli-seconds.

What would cause such a large difference is hard to guess. The above is on my windows laptop, with very little CPU or network activity.

One possible reason: During the first run the OS buffers (file system cache) will be populated with the data you read, and this data will be available to the second run via the cache. This is a common issue with database tests. There is a way to clear the OS buffers used by the file system before each test, but I don't have it handy, I'll let you know if I find it.

This is what I was doing on Linux, although it's been a while since I've used it:

sync; echo 1 > /proc/sys/vm/drop_caches

In this test there is no permanent database, it is just "mocked" as a "memory file". So it isn't that. Also, I am running the test many times. I actually run three tests in parallel, and what is interesting is that the times correlate to some extent, so if one test runs a bit slower, so do the others.

One thing I am wondering about, this: What Is Intel® Turbo Boost Technology? - Intel

"CPUs don’t always need to run at their maximum frequency. Some programs are more dependent on memory to run smoothly, while others are CPU-intensive. Intel® Turbo Boost Technology is an energy-efficient solution to this imbalance: it lets the CPU run at its base clock speed when handling light workloads, then jump to a higher clock speed for heavy workloads."

My processor is "Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz 2.71 GHz".

It apparently does have "boost".

" Intel® Turbo Boost Technology 2.0 Frequency
3.10 GHz
Processor Base Frequency
2.50 GHz"

https://www.intel.com/content/www/us/en/products/sku/95443/intel-core-i57200u-processor-3m-cache-up-to-3-10-ghz/specifications.html

Apparently I can try disabling it...

"Though you can also disable Intel® Turbo Boost Technology from the BIOS, it’s not recommended to do so unless you’re troubleshooting specific issues or trying to collect consistent performance measurements."

[ Edit: I just tried, but couldn't find it in my BIOS settings ]

Well, I tried running the tests one at a time, and running each test 100 times. A lot of the variability disappeared. Sample output below:

C:\Users\ano31\Rust\RustDB>cargo test bench::rustdb_direct_test --release -- --nocapture
    Finished release [optimized] target(s) in 0.06s
     Running unittests src\lib.rs (target\release\deps\rustdb-4a3e5ed9d6a4fe2e.exe)

running 1 test
rustdb direct test took 170 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 197 milli-seconds
rustdb direct test took 154 milli-seconds
rustdb direct test took 187 milli-seconds
rustdb direct test took 162 milli-seconds
rustdb direct test took 167 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 155 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 154 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 156 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 154 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 149 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 165 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 161 milli-seconds
rustdb direct test took 177 milli-seconds
rustdb direct test took 164 milli-seconds
rustdb direct test took 159 milli-seconds
rustdb direct test took 155 milli-seconds
rustdb direct test took 155 milli-seconds
rustdb direct test took 154 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 156 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 156 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 154 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 161 milli-seconds
rustdb direct test took 154 milli-seconds
rustdb direct test took 165 milli-seconds
rustdb direct test took 154 milli-seconds
rustdb direct test took 155 milli-seconds
rustdb direct test took 155 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 191 milli-seconds
rustdb direct test took 164 milli-seconds
rustdb direct test took 160 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 180 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 149 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 154 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 155 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 151 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 152 milli-seconds
rustdb direct test took 153 milli-seconds
rustdb direct test took 150 milli-seconds
rustdb direct test took 151 milli-seconds
test bench::rustdb_direct_test ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 10 filtered out; finished in 15.59s

I think maybe the thing to do is to throw out say the 50 "worst" times (on the grounds that something external must have delayed the test) and average the remaining 50....

Results from doing this:

sqlite_test average time=376
rustdb_test average time=345
rustdb_direct_test average time=147