Skip to content

Input System

Jump to bottom
pinjinx edited this page Oct 23, 2026 · 2 revisions

The input system allows a phone, tablet, or browser to function as a wireless trackpad, keyboard, and virtual gamepad for the host computer. It converts browser touch gestures and on-screen controls into native operating system input events while maintaining sub-8ms latency and platform-independent behavior.

From a user's perspective, every interaction feels immediate and natural. Internally, Rein recognizes gestures, transports them over specialized WebRTC DataChannels, validates and sanitizes incoming messages, throttles high-frequency motion, and injects native input events using cross-platform @imxade/inject drivers (Linux, macOS, Windows).

Input Pipeline

Every input event follows the same processing pipeline.

Untitled-2026-08-02-1922 (5)

Gesture Recognition (useTrackpadGesture)

Gesture recognition is handled by useTrackpadGesture (src/hooks/useTrackpadGesture.ts).

The hook listens for browser touch events (touchstart, touchmove, touchend, and touchcancel) and converts them into higher-level input actions that can be transmitted to the host.

Internally it maintains a map of active touch points and continuously updates gesture state as fingers are added, moved, or removed.

Supported Trackpad Gestures

Fingers Gesture Generated Event Channel
1 Move Relative cursor move (dx, dy) input-unordered
1 Tap (within 250ms) Left click (button: "left", press: true/false) input-ordered
2 Tap (within 250ms) Right click (button: "right") input-ordered
3 Tap (within 250ms) Middle click (button: "middle") input-ordered
1 Tap + immediate move Dragging (holds left button while moving) input-unordered
2 Pan Two-finger scroll (dx, dy) input-unordered
2 Pinch Pinch-to-zoom (delta) input-unordered
Scroll mode 1 finger Directional scroll with axis lock input-unordered

Scroll Axis Lock

When one-finger scroll mode is enabled, Rein prevents accidental diagonal scrolling by locking movement to the dominant axis:

if (absDx > absDy * axisThreshold)
    dy = 0
else if (absDy > absDx * axisThreshold)
    dx = 0

The default threshold is 2.5, meaning one direction must be significantly stronger before the other axis is ignored.

Movement Thresholds

Small unintended movements are ignored before a gesture begins:

Fingers Threshold
1 10 px
2 15 px
3+ 15 px

Drag Detection

Dragging is implemented by converting a quick double-tap-and-hold into an active drag state:

Untitled-2026-08-02-1922 (7)

Absolute Pointer / Touchscreen Mode

In addition to relative trackpad mode, Rein supports direct touchscreen tap-to-click via rein_mouse_mode:

  1. Letterbox-Aware Coordinate Normalization (toNormalisedRatio): The video screen mirror renders with CSS object-fit: contain. When a user taps anywhere on the screen, toNormalisedRatio() calculates the exact content rectangle (accounting for horizontal pillarboxing or vertical letterboxing) and computes normalized x, y ∈ [0, 1] ratios relative to the desktop display.
  2. absoluteClick Event: Sent over input-ordered: 
    {
  type: "absoluteClick",
  button: "left",
  press: true,
  x: 0.452,
  y: 0.781
}
  3. Host Injection: The host InputHandler multiplies normalized ratios by configured screen dimensions (absX = msg.x * screenWidth, absY = msg.y * screenHeight), calls injectMouseAbsolute(absX, absY) to warp the cursor instantly without relative acceleration artifacts, and fires injectMouseButton(button, press).
  4. Physical Mouse Pointer Lock (useMouseInput): When using a desktop browser to control another desktop, useMouseInput.ts captures pointer lock on mouse click, locks the cursor, routes raw movementX/movementY deltas, passes native mouse button states, wheel events, and physical keyboard shortcuts directly to the remote host.

Virtual Gamepad System

Rein includes a built-in virtual gamepad controller accessible at /gamepad (src/routes/gamepad.tsx).

Gamepad UI & Layout (GamepadOverlay.tsx, GamepadLayoutSettings.tsx)

  • Dual Analog Thumbsticks: Left stick (ls) and right stick (rs) with smooth touch tracking and deadzone handling.
  • D-Pad: Directional pad (dpad-up, dpad-down, dpad-left, dpad-right).
  • Action Buttons: Standard Xbox-style diamond buttons (a, b, x, y).
  • Shoulder Buttons & Triggers: Left/Right bumpers (lb, rb) and digital triggers (lt, rt).
  • Menu Buttons: start and select.
  • Customizable Layout: Opacity, scale, and thumbstick positions can be adjusted in the Gamepad Settings tab and saved to localStorage (rein_gamepad_layout).

Gamepad Event Routing

  • Digital Buttons: { type: "gamepad", button: GamepadButtonId, press: boolean } sent via input-ordered (ordered: true).
  • Analog Axes: { type: "gamepad-axis", axis: "ls" | "rs", ax: number, ay: number } sent via input-unordered (maxRetransmits: 0).
    • Axis coordinates are normalized between -1.0 and +1.0.
    • Throttled independently per stick at 8ms.
    • Releasing a stick back to neutral (0, 0) immediately cancels any pending throttle timer and dispatches instantly to guarantee the character stops moving immediately.

Transport Layer (Dual DataChannels)

Rein balances responsiveness and reliability using two dedicated WebRTC DataChannels:

  • input-unordered channel (ordered: false, maxRetransmits: 0): Used for high-frequency continuous motion: move, scroll, zoom, touch, gamepad-axis. Dropping an occasional intermediate motion packet is vastly preferable to stalling subsequent packets with Head-of-Line (HoL) blocking.
  • input-ordered channel (ordered: true): Used for discrete state transitions and commands that must never be lost or reordered: click, absoluteClick, key, text, combo, copy, paste, gamepad, ping/pong.

ConnectionProvider automatically routes each event to the correct channel and provides seamless fallback if a channel is temporarily unavailable.

Latency Measurement (Ping/Pong)

To measure connection latency, ConnectionProvider exchanges ping and pong messages over the reliable DataChannel:

Client (ConnectionProvider)
   │ { type: "ping", timestamp: Date.now() }
   ▼
Server (WebRTCManager)
   │ { type: "pong", timestamp: timestamp }
   ▼
Client

The client measures RTT (Date.now() - timestamp) every 2 seconds and reports the value to POST /api/debug/report-latency every 10 seconds.

Host InputHandler (src/server/InputHandler.ts)

InputHandler processes every incoming input message on the host before delegating to the native operating system injector:

  1. Validation & Sanitization:
    • Caps text length to MAX_TEXT_LENGTH = 10000.
    • Clamps motion coordinates dx, dy, delta to [-2000, 2000].
    • Clamps gamepad axes ax, ay to [-1, 1].
    • Clamps normalized absolute coordinates x, y to [0, 1].
    • Validates mouse buttons and combo key arrays.
  2. Rate Limiting & Throttling (8ms):
    • Uses an 8ms leading-edge throttle with trailing timer flush for move, scroll, and gamepad-axis.
    • Prevents OS input event queue saturation while ensuring the final position is always delivered.
  3. Motion Acceleration Curve (applyMotion):
    • Computes non-linear acceleration for trackpad cursor movement based on velocity: 
      ax = Math.sign(dx) * Math.pow(Math.abs(dx), ACCEL_EXPONENT) * ACCEL_FACTOR * sensitivity
  4. Safety & Cleanup:
    • Automatically releases mouse buttons if an injection call throws, preventing stuck clicks.
    • Clears pending motion and stick timers on session destroy.

Cross-Platform Input Injection Layer

After sanitization and throttling, InputHandler dispatches commands to the platform driver implementing PlatformInjector. All native input injection is built on @imxade/inject, providing lightweight low-level system input primitives without C++ compilation requirements:

Platform Injector Class Module Native Injection Mechanism
Linux LinuxInputInjector @imxade/inject/linux /dev/uinput devices (mouse, absolute pointer, keyboard, multitouch, Xbox 360 gamepad)
macOS MacInputInjector @imxade/inject/mac CoreGraphics CGEvent system event injection & IOHIDUserDevice virtual gamepad
Windows WindowsInputInjector @imxade/inject/windows Win32 SendInput, Synthetic Pointer API (POINTER_TYPE_INFO), & ViGEmBus Xbox 360 controller

For complete implementation details on virtual devices, permissions, and report structures, see Platform Drivers.

System Clipboard Synchronization

Rein supports bidirectional clipboard sharing between the client and host:

  1. HTTP Clipboard API (POST /api/clipboard/copy & POST /api/clipboard/paste):
    • /api/clipboard/copy: Synthesizes Ctrl+C (or Cmd+C on macOS), waits 100ms for the OS clipboard to populate, and reads plain text using native tools (pbpaste on macOS, PowerShell Get-Clipboard on Windows, wl-paste/xclip/xsel on Linux).
    • /api/clipboard/paste: Writes client text into the host system clipboard via stdin without command-injection vulnerabilities, then synthesizes Ctrl+V (or Cmd+V on macOS).
  2. DataChannel Copy/Paste:
    • The trackpad control bar provides one-tap Copy and Paste buttons which send { type: "copy" } and { type: "paste" } shortcuts directly over input-ordered.

Input Message Specification

export interface TouchContact {
  id: number
  x: number
  y: number
  state: "down" | "move" | "up"
}

export type MouseButton = "left" | "right" | "middle"

export type GamepadButtonId =
  | "a" | "b" | "x" | "y"
  | "lb" | "rb" | "lt" | "rt"
  | "start" | "select"
  | "dpad-up" | "dpad-down" | "dpad-left" | "dpad-right"
  | "ls" | "rs"

export type InputMessage = {
  dx?: number
  dy?: number
  config?: Partial<InputConfig>
  key?: string
  keys?: string[]
  text?: string
  delta?: number
  contacts?: TouchContact[]
  axis?: "ls" | "rs"
  ax?: number
  ay?: number
} & (
  | { type: "move" }
  | { type: "paste" }
  | { type: "copy" }
  | { type: "click"; button: MouseButton; press: boolean }
  | { type: "absoluteClick"; button: MouseButton; press: boolean; x: number; y: number }
  | { type: "scroll" }
  | { type: "key" }
  | { type: "text" }
  | { type: "zoom" }
  | { type: "combo" }
  | { type: "touch" }
  | { type: "gamepad"; button: GamepadButtonId; press: boolean }
  | { type: "gamepad-axis" }
)

Wiki

Guides

API

Clone this wiki locally