How can I Capture Global Inputs with XDG Desktop Portal without Restricting the Mouse?

Hi, I am building a Linux-exclusive macro software that needs to capture global inputs even if the window is minimized and the application is a Flatpak app.

I found this Portal. But I realized that it only captures when the pointer is inside the barrier. And even worse, it locks up the mouse! Is there a way to make the mouse not lock up.

If you don't know what I am talking about try running the script below:
I got this script from the ashpd documentation

use std::{collections::HashMap, os::unix::net::UnixStream, sync::OnceLock, time::Duration};

use ashpd::desktop::input_capture::{Barrier, BarrierID, Capabilities, InputCapture};
use futures_util::StreamExt;
use reis::{
    ei::{self, keyboard::KeyState},
    event::{DeviceCapability, EiEvent, KeyboardKey},
};

#[allow(unused)]
enum Position {
    Left,
    Right,
    Top,
    Bottom,
}

static INTERFACES: OnceLock<HashMap<&'static str, u32>> = OnceLock::new();

async fn run() -> ashpd::Result<()> {
    let input_capture = InputCapture::new().await?;

    let (session, _cap) = input_capture
        .create_session(
            None,
            Capabilities::Keyboard | Capabilities::Pointer | Capabilities::Touchscreen,
        )
        .await?;

    // connect to eis server
    let fd = input_capture.connect_to_eis(&session).await?;

    // create unix stream from fd
    let stream = UnixStream::from(fd);
    stream.set_nonblocking(true)?;

    // create ei context
    let context = ei::Context::new(stream)?;
    context.flush().unwrap();

    let (_connection, mut event_stream) = context
        .handshake_tokio("ashpd-mre", ei::handshake::ContextType::Receiver)
        .await
        .expect("ei handshake failed");

    let pos = Position::Left;
    let zones = input_capture.zones(&session).await?.response()?;
    eprintln!("zones: {zones:?}");
    let barriers = zones
        .regions()
        .iter()
        .enumerate()
        .map(|(n, r)| {
            let id = BarrierID::new((n + 1) as u32).expect("barrier-id must be non-zero");
            let (x, y) = (r.x_offset(), r.y_offset());
            let (width, height) = (r.width() as i32, r.height() as i32);
            let barrier_pos = match pos {
                Position::Left => (x, y, x, y + height - 1), // start pos, end pos, inclusive
                Position::Right => (x + width, y, x + width, y + height - 1),
                Position::Top => (x, y, x + width - 1, y),
                Position::Bottom => (x, y + height, x + width - 1, y + height),
            };
            Barrier::new(id, barrier_pos)
        })
        .collect::<Vec<_>>();

    eprintln!("requested barriers: {barriers:?}");

    let request = input_capture
        .set_pointer_barriers(&session, &barriers, zones.zone_set())
        .await?;
    let response = request.response()?;
    let failed_barrier_ids = response.failed_barriers();

    eprintln!("failed barrier ids: {:?}", failed_barrier_ids);

    input_capture.enable(&session).await?;

    let mut activate_stream = input_capture.receive_activated().await?;

    loop {
        let activated = activate_stream.next().await.unwrap();

        eprintln!("activated: {activated:?}");
        loop {
            let ei_event = event_stream.next().await.unwrap().unwrap();
            eprintln!("ei event: {ei_event:?}");
            if let EiEvent::SeatAdded(seat_event) = &ei_event {
                seat_event.seat.bind_capabilities(&[
                    DeviceCapability::Pointer,
                    DeviceCapability::PointerAbsolute,
                    DeviceCapability::Keyboard,
                    DeviceCapability::Touch,
                    DeviceCapability::Scroll,
                    DeviceCapability::Button,
                ]);
                context.flush().unwrap();
            }
            if let EiEvent::DeviceAdded(_) = ei_event {
                // new device added -> restart capture
                break;
            };
            if let EiEvent::KeyboardKey(KeyboardKey { key, state, .. }) = ei_event {
                if key == 1 && state == KeyState::Press {
                    // esc pressed
                    break;
                }
            }
        }

        eprintln!("releasing input capture");
        let (x, y) = activated.cursor_position().unwrap();
        let (x, y) = (x as f64, y as f64);
        let cursor_pos = match pos {
            Position::Left => (x + 1., y),
            Position::Right => (x - 1., y),
            Position::Top => (x, y - 1.),
            Position::Bottom => (x, y + 1.),
        };
        input_capture
            .release(&session, activated.activation_id(), Some(cursor_pos))
            .await?;
    }
}

Use dependencies from this toml:

[package]
name = "bloons-td-no-punjabi-virus-free-vbucks-gahfrtgr-help-pls"
version = "0.1.0"
edition = "2021"

[dependencies]
tokio = { version = "1.43.0", features = ["full"] }
ashpd = { version = "0.10.2", features = ["tokio", "raw_handle", "wayland"] }
reis = { version = "0.4.0", features = ["tokio"] }
futures-util = "0.3.31"

As you can see that it locks up the mouse, is there another way to capture global input in Flatpak and Wayland?

Thank you for reading.

If I understand the documentation correctly, the point of the barrier is to only capture when you are inside the barrier. And once the input is captured all input events are delivered to the program until the program tells the system not to capture inputs anymore.

If you don't want to capture the pointer, but only the keyboard, remove Capabilities::Pointer | Capabilities::Touchscreen.

Also the Global Shortcuts portal may be more appropriate. It allows you to intercept just specific keyboard inputs rather than capturing everything.

1 Like

Thanks I will look at this in the morning. But don't I still need to set, the barriers to allow input capture, which means physically dragging the mouse to the corner.

I tried the GlobalShortcuts XDG Desktop Portal but it has the limitation that you need to have either Ctrl, Alt, super which is not ideal.

Can I just directly connect to EIS interface? If it can, will it require sudo?

Looks like you are right. From the docs:

Input capturing is an asynchronous operation using “triggers”. An application sets up a number of triggers but it is the compositor who decides when the trigger conditions are met. Currently, the only defined trigger are pointer barriers: horizontal or vertical “lines” on the screen that should trigger when the cursor moves across those lines. See org.freedesktop.portal.InputCapture.SetPointerBarriers.

There is currently no way for an application to activate immediate input capture.

I think the Remote Desktop portal is your best bet. It seems like you can use the SelectDevices method to control just the keyboard, start the session using Start, then use the ConnectToEIS method to get an EIS handle to receive inputs from and then use NotifyKeyboardKeycode/NotifyKeyboardKeysym to forward any unprocessed keyboard input.

I think the Remote Desktop portal is your best bet. It seems like you can use the SelectDevices method to control just the keyboard, start the session using Start, then use the ConnectToEIS method to get an EIS handle to receive inputs from and then use NotifyKeyboardKeycode/NotifyKeyboardKeysym to forward any unprocessed keyboard input.

This won't work because ConnectToEIS method from RemoteDesktop from XDG Desktop Portal only allows emulating input, it doesn't allow capturing input.

I have another last thing I can try before just building a typical ydotool wrapper. There's a script on KDE for controlling the desktop environment called kwinscript. Since the compositor is the only one that can get input, I think maybe I can make my app build a kwinscript and give it to kwin to run it. The kwin script will be very simple, It only needs to register the keybind and send a dbus output to my app. The best thing of doing it this way is that is still a flatpak but it there will be troubles when the user is using another DE.

At the bottom we have a QML script that is a declarative language called shortcut.qml that will be made into a zip with .kwinscript extention:

import QtQuick
import org.kde.kwin

Item {

    ShortcutHandler {
        name: "Macro Test"
        text: "Get A Print B"
        sequence: "A"
        onActivated: {
        func: callDBus() // <-- Here is wrong idk how to use this one yet.
        }
    }
}

edit: This ctully works I cnt spell the binded keybind rn. (17:07PM)

I also found this thread for managing kwinscripts via the command line.

I am going to see how to implement on GNOME desktop later. I am working on the KDE solution first. It might take long cuz I don't know any javascript at all but kwinscript uses javascript.

I am stumped. Since you can use kwinscript to call dbus, there's no point of creating an app in the first place. Because you can just call RemoteDesktop XDG Desktop Portal via the kwinscript. I just wasted my time. There's no need for scripting an agent for managing separate macro instances. This is kinda sad because I wasted a lot of time. I wonder is it possible to do it with GNOME tho.

So the solution is:
Don't build the app.
The user will need to build a GNOME extension or Kwinscipt.