Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

How unsupervised trajectory synthesis is breaking the data bottleneck for cross-platform GUI automation at enterprise scale

Introduction: The GUI Automation Paradox and Why Data Scarcity is the Bottleneck

The enterprise software landscape is trapped in a paradox: while multimodal large language models (LLMs) have achieved human-level performance on complex reasoning tasks, 70% of enterprise software interactions still rely on manual GUI-based workflows McKinsey & Company. This disconnect represents a significant automation opportunity, yet the path to realizing it is blocked by a fundamental constraint—data scarcity in GUI agent training. Existing datasets like RICO (~66k trajectories) and Mind2Web (~2k trajectories) are orders of magnitude too small to train generalized agents capable of operating across web, mobile, and desktop platforms arXiv (RICO and Mind2Web papers). The Video2GUI framework emerges as a solution to this bottleneck, using unsupervised learning techniques to extract interaction trajectories from raw screen recordings, reducing reliance on labeled data arXiv.

The Rise of Multimodal LLMs and the GUI Agent Opportunity

The convergence of multimodal LLMs (e.g., GPT-4V, Gemini, LLaVA) and foundation models for UI understanding (e.g., Microsoft’s UI-T5, Google’s Screen Understanding models) has unlocked new capabilities in GUI automation Microsoft Research. These models can:

• Parse hierarchical UI structures (e.g., DOM trees, Android view hierarchies) from raw pixels.
• Ground natural language instructions (e.g., "Book a flight to Berlin") to specific UI elements (e.g., a "Departure City" dropdown).
• Plan multi-step trajectories (e.g., "Log in → Navigate to settings → Update profile picture") across heterogeneous platforms.

Despite these advances, deployment remains limited to narrow, high-value use cases (e.g., customer support bots, data entry automation) due to the lack of diverse, large-scale training data. The Physical AI Stack provides a useful lens for understanding this gap:

• SENSE: GUI agents must perceive dynamic UI states (e.g., modal dialogs, loading spinners) across resolutions, aspect ratios, and platforms.
• REASON: Agents must generalize across UI paradigms (e.g., web forms vs. mobile swipe gestures) and adapt to unseen applications.
• ORCHESTRATE: Enterprise deployments require coordination across edge devices (e.g., RPA bots) and cloud inference (e.g., LLM-based planning).

Current datasets fail to address these layers holistically. For example:

• RICO focuses on mobile apps and lacks web/desktop coverage ACM Digital Library.
• Mind2Web is web-only and limited to 2,000 trajectories, insufficient for cross-platform generalization arXiv.
• Manual curation (e.g., crowdsourcing) is cost-prohibitive at scale, with annotation costs exceeding $0.50 per trajectory for complex workflows arXiv (Video2GUI Paper).

The Economic Imperative: Why GUI Automation is a Critical Opportunity

Enterprises spend $1.8 trillion annually on digital transformation, yet 70% of software interactions remain manual McKinsey & Company. GUI automation targets this inefficiency by:

1. Reducing operational costs: Automating repetitive tasks (e.g., invoice processing, customer onboarding) can cut labor costs.
2. Improving accuracy: GUI agents eliminate human errors in data entry (e.g., typos, misclicks).
3. Enabling 24/7 operations: Unlike human workers, agents can operate continuously (e.g., overnight batch processing).

However, the lack of generalized agents forces enterprises to:

• Build bespoke bots for each application (e.g., one for SAP, another for Salesforce), increasing maintenance overhead.
• Rely on brittle selectors (e.g., XPath, CSS) that break with UI updates.
• Limit automation to low-complexity tasks (e.g., form filling) due to poor generalization.

The Data Scarcity Bottleneck: Why Existing Datasets Fail

To train a generalized GUI agent, a dataset must satisfy three criteria:

1. Scale: Millions of trajectories to cover edge cases (e.g., error states, rare UI patterns).
2. Diversity: Cross-platform coverage (web, mobile, desktop) and cross-domain tasks (e.g., e-commerce, healthcare).
3. Realism: Trajectories must reflect real-world noise (e.g., network latency, partial occlusions).

Existing datasets fall short on all three fronts:

Key failure modes:

• Manual annotation is unscalable: Labeling a single trajectory (e.g., "Book a flight") can take 5-10 minutes, costing $0.50-$1.00 per sample arXiv (Video2GUI Paper).
• Synthetic data is unrealistic: Rule-based generators (e.g., WebGUM) produce deterministic, noise-free trajectories that fail to generalize to real-world UIs.
• Platform silos: Agents trained on web data fail to transfer to mobile/desktop (e.g., a "swipe" gesture has no web equivalent).

Video2GUI: Unsupervised Trajectory Synthesis as a Force Multiplier

Video2GUI reframes the problem by treating raw screen recordings as a data source rather than relying on labeled trajectories. Its pipeline consists of four stages:

1. Capture: Record raw screen recordings (e.g., desktop sessions, mobile app usage) using tools like OpenCV and FFmpeg.
2. Parse: Extract UI elements (e.g., buttons, text fields) and actions (e.g., clicks, scrolls) using:
   • OCR (Tesseract) for text extraction.
   • Object detection (YOLO, Faster R-CNN) for UI elements.
   • Contrastive learning (SimCLR) to cluster similar UI states.
3. Synthesize: Generate synthetic trajectories by:
   • Sampling plausible action sequences (e.g., "Click login → Enter credentials → Submit").
   • Augmenting with noise (e.g., random delays, partial occlusions) to improve realism.
4. Validate: Filter trajectories using:
   • Heuristics (e.g., "A login form must have a submit button").
   • Human-in-the-loop (e.g., crowdsourced quality checks for a subset of samples).

Key innovations:

• Unsupervised learning: Eliminates the need for manual labels by leveraging self-supervised contrastive learning (SimCLR) to group similar UI states arXiv (Video2GUI Paper).
• Cross-platform generalization: Trajectories are synthesized for web, mobile, and desktop using a unified action space (e.g., "CLICK", "TYPE", "SCROLL").
• Cost efficiency: Reduces annotation costs by 50-70% compared to manual labeling arXiv (Video2GUI Paper).

Benchmarking Video2GUI Against Existing Datasets

Video2GUI’s 1.2M synthetic trajectories dwarf existing datasets in scale and diversity:

Performance impact:

• Agents trained on Video2GUI achieve 20-30% higher task completion rates in cross-platform scenarios (e.g., "Book a hotel" across web and mobile) arXiv (Video2GUI Paper).
• Generalization to unseen apps: Video2GUI-trained agents perform 15-25% better on zero-shot tasks (e.g., navigating a new CRM system) compared to agents trained on RICO or Mind2Web.

The Physical AI Stack: Deploying Video2GUI in Enterprise Workflows

Video2GUI aligns with the Physical AI Stack by addressing each layer’s requirements:

EU AI Act Compliance:

• Video2GUI’s on-premise deployment ensures data sovereignty (e.g., screen recordings never leave the enterprise network).
• Transparency: Synthetic trajectories are auditable (e.g., validation logs, human-in-the-loop checks).
• Risk mitigation: High-risk use cases (e.g., healthcare, banking) can be isolated from low-risk tasks (e.g., customer support) European Parliament (EU AI Act).

Core Concepts: Interaction Trajectories, Multimodal Grounding, and the Physical AI Stack

1. Formal Definition of Interaction Trajectories

An interaction trajectory in GUI automation is a temporally ordered sequence of tuples $\tau = {(s_t, a_t, r_t, s_{t+1})}_{t=1}^T$, where:

• $s_t \in \mathcal{S}$ is the state at time $t$, represented as a multimodal tensor combining:
  • Visual state: A $H