Computer Vision in Rust?


#1

Hi,

I’m currently working on computer vision algorithms, more specifically direct visual odometry (similar to DSO). As far as I know all these algorithms are implemented in C++ (or Matlab). I have the feeling that this project could be a good opportunity to try out Rust. I love the idea of having a very fast language, with a much better type system than C++.

To my knowledge, the requirements of such a project are:

  • Reading and showing images / videos ( / streams).
  • Manipulation of images as matrices / tensors.
  • Linear algebra with those matrices / tensors:
    • Standard linear algebra operations (additions, multiplications, decompositions, …).
    • Specific linear algebra of rigid body motions, similarities, projective geometry, …
  • Linear and non-linear solvers and optimization, like Gauss-Newton optimization.
  • Graph optimization (similar to g2o).
  • Appearance-based large-scale loop closure detection (similar to openFABMAP).
  • Manipulation of point clouds.
  • 3D scene visualization:
    • Visualization of point clouds.
    • Visualization of simple shapes, lines, meshes.
    • Scene navigation (being able to zoom, rotate, …).

That makes a lot of requirements and I might have forgotten some. I’d rather do the project entirely in one language, either C++ or Rust but no mix (or max on library). So my questions are:

  • Do you think the Rust ecosystem is mature enough for a project like this? (available / missing libraries)
  • Are there hard stoppers, caveats, warnings I should be aware of?
  • Are there other computer vision projects in Rust?

Thanks in advance for any honest answer!


#2

How secure do you need your library to be?

I suspect the ease of development in C++, the amount of mature libraries available in C++ ecosystem heavily outweigh the safety you get in Rust.

There are the typical vulnerabilities related to memory errors in C++ libraries(e.g. OpenCV), leading to arbitrary code execution, DoS, etc, but their initial development began when security was not a major concern. If you are writing your algorithms from scratch in C++ along with help of static and dynamic analyzers, then you probably will have easier time developing than in Rust and achieve roughly same level of safety depending on how critical your library/system needs to be.


#3

Safety is not my main concern. What attracts me most is the expressiveness of the type system (ADT) and the ownership model. And I’d like to know how much I can reuse (available libraries) and how much I’d have to recode.

So far, I’ve identified:

  • PistonDevelopers/image and pcwalton/rust-media for image / video manipulation
  • rust-ndarray and nalgebra for matrices manipulation, projective geometry, linear solvers
  • bluss/petgraph for expression of graphs
  • l3ck/rust-3d or aepsil0n/acacia for point clouds
  • For visualization:
    • PistonDevelopers/conrod for GUI visualization
    • sebcrozet/kiss3d for graphics engine (visualize point clouds, cameras, etc.).
    • gfx for low-level graphics

I didn’t find any library for:

  • non-linear optimization
  • optimization over graphs
  • loop-closure detection (image appearance algorithm)

Any feedback on these libraries about maturity, maintenance, inter compatibility?


#4

I think a lib like opencv is important to computer vision in Rust. Yes, there is a opencv binding to Rust, which however has not updated for two years, and the document is not compariable to its origin C++ counterpart.

The data i/o lib is also far from usable. For example, many scientific data are stored as hdf5. The rust hdf5 lib is available, however the document is not very friendly, and it has not been updated for two years.


#5

The corresponding reddit post for reference.


#6

pcwalton/rust-media is not maintained. I recommend using gstreamer wrapper instead.


#7

@kornel I’d really like to avoid c/c++ wrappers but if there is no choice thanks for the reference. Are there alternatives in Rust? Are there libraries with high level abstractions like what you’d do in OpenCV?

cap = cv2.VideoCapture(0);
frame = cap.read();

#8

I’m not aware of any production-ready video codecs in Rust, so avoiding C wrappers will be very limiting.

C wrappers usually do add high level abstractions and a level of safety on top of the C APIs.


#9

I just watched this video from FOSDEM 18 about GStreamer and Rust so I’m just adding the reference here: