Dars-dap and hidefix (concurrent DAP/2 server and concurrent HDF5 reader)

Hi,

I am working on a rust dap server: dars. It allows access to HDF5 and Netcdf(4) files through the DAP/2 protocol (much used in science). HDF5 (and netcdf, which uses HDF5 in newer versions) turns out to be not thread-safe at all (even across different files or datasets). This limited concurrent performance, and I therefore started a (thread-safe) rust HDF5 reader: hidefix which performs quite well, on par or better for sequential reads -- but far better for multi-threaded reads. It is not meant to replace rust-hdf5 or rust-netcdf, but maybe be used together with these. Both of these packages are experimental, and although basic functionality is there, incomplete. dars is not on crates.io, because it needs some patches to HDF5 source.

Some simple benchmarks between dars and the two main existing server implemations: thredds and hyrax. Latency comparison was difficult to do because of large difference in performance.


Notably, the server load is also fairly different: