Are there any video capture libraries out there?


#1

I’m working on a screen- and camera-capture library right now (for Windows/macOS/Linux) but forgot to check whether something like that already exists! :grin: The closest I could find are rust-media and some GStreamer bindings.


#2

I made an analysis on this topic a couple months ago, hope this helps.

Camera Capture

  • rscam (Linux) - quite versatile and easy to use.
  • escapi (Windows) - I’ve recently pushed a pull requests which adds Rust support there, the library is a bit limited and quite restrictive, but usable though.
  • camera_capture - just a thin wrapper around rscam and escapi, but I personally did not like it, so I preferred to write my own.

Screen Capture

  • x11cap (Linux)
  • dxgcap (Windows)
  • captrs - a thin wrapper around x11cap and dxcap.

#3

By the way, I also thought about creation of such library, in general it would be nice to have a crossplatform library (if possible) to handle screen capture and camera capture (it makes sense to make them as 2 separate crates I guess). If I were going to write a new screen sharing library, I would reuse the libraries mentioned in my previous comment and tried to extend them, for instance dxgcap looks like a good thing for Windows, but there are many things that might be improved there (I opened several issues there, but the initial author is busy and stopped development for the library, so maybe it makes sense to take it over and/or create a new library and use the similar approach). dxgcap is based on Desktop Duplication API which is a good thing if you want to have a fast screen capturing.

I have not seached for Mac screen capturing libaries, but it seems that there are no good libraries for that. For Mac I would suggest to use CGDisplayStream API as it is the fastest way to capture the screen there.

Regarding the camera capture - I would use rscam for Linux (it seems to be quite functional at the moment and I did not have problems with it so far) and I would fully rewrite escapi from scratch to make it pure Rust library, the current C++ version has many problems (and the code quality is not that good there) and quite a few restrictions.


#4

Thanks for the advice! :smile: This is the list of libraries that need interfacing with so far:

Linux:

  • xcb-shm
  • There seems to be no unified interface for Wayland yet. VLC mysteriously has a Wayland recorder. :confused:
  • v4l2, probably through rscam.

macOS:

  • CGDisplayStream looks a lot better than AVFoundation for screen capture. Thanks!
  • AVFoundation seems to be the only option for video capture. Any idea what it’s built on?

Windows:

  • DXGI (Desktop Duplication API)
  • Media Foundation

#5

Unfortunately I have not researched camera capturing APIs for Mac. Usually I use OBS Studio as a reference, they have implemented a high performance and reliable screen and camera capturing and the code is well structured (although it’s not well documented in many parts).


#6

I have used battle-tested OpenCV for just this purpose (on Windows). Using something Rust would of course be better, but OpenCV as a lib has the advantage of being cross-platform. FWIW, compared to anything else I have tried it was far superior. Note: I have no Rust experience. Uses C-bindings.


#7

Mystery behind VLC on Wayland it that it uses weston-screenshooter protocol. If you’d like to see example in Rust check this: https://github.com/perceptia/perceptia/blob/master/perceptia/perceptiactl/screenshot.rs


#8

I considered OpenCV as an option, but there are some reasons why I think it’s not the best solution in a long run:

  1. The one has to ensure that OpenCV uses Media Foundation for Windows as a backend.
  2. I’m not sure that it gives us the same flexibility in configuration as we would have if we could use Media Foundation directly.
  3. OpenCV is quite a big libary, carrying the whole library for camera capturing only seems to be an overkill. Moreover it would require either a build script which builds the whole library every time the crate is being built or having multiple versions of the library for all supported platforms (and placing them along other files in the repository).
  4. Last time when I checked the Rust bindings, it did not even compile, there were quite a few issues, so I had to switch to ESCAPI back then.

#9

Progress:

  • macOS screen capture working now, there’s an ffplay example at quadrupleslap/scrap. :smile:
  • X11 with xcb-shm done, will push tomorrow afternoon.

Remaining:

  • DXGI/Windows support.
  • Figuring out how to combine all of these into one interface, which so far has turned out to be painful. :slight_frown:

It’s hard because the X11 version just has a get_frame function that you call to get the next frame, but the macOS version has this strange interface where you pass it a function that gets called whenever the OS feels like it. Any ideas/help would be appreciated, because I really don’t want to just use an Arc<Mutex<Frame>> for the macOS version.

Edit: Also, OpenCV sounds like an interesting alternative, but it’s a huge framework with a lot more than I need/necessarily want.


#10

I hope it’s a temporary solution :slight_smile:

Actually a bit similar concept is present in Windows (although it’s implemented in a slightly different manner), if we’re talking about internals. So regarding the Mac, I think there are not too many options here, probably Arc would do the job, as you mentioned. I’ve just checked the OBS source code again, they indeed use a mutex, but they lock it only for a short period of time in order to change the pointers to a corresponding surface, probably you can do something like that.