MMSkill-RL

Overview

The question

Given a task, a screenshot, and history, a visual GUI agent must decide: act directly, retrieve a procedural text skill, retrieve a visual exemplar, or combine both, and then turn whatever it retrieved into a correct GUI action. MMSkill-RL treats this four-way routing decision (DIRECT / QUERY_TEXT / QUERY_IMAGE / QUERY_BOTH) as the object of learning.

Method in one line: minimal SFT warm start for format and skill-conditioned action, then outcome-based RL so the weak model learns when to query, instead of imitating a strong model's choices.

Proposal

Contribution targets

1. A clean multimodal GUI skill pool — deduplicated, provenance-tracked, frozen.
2. A skill-sensitive benchmark and oracle-training signal — every step labeled by which conditions (direct/text/image/both) a strong model can solve it under.
3. Evidence that explicit skill use reduces none/uncovered and lifts weak-model action correctness.
4. An end-to-end skill-use policy — the weak model learns DIRECT / QUERY_TEXT / QUERY_IMAGE / QUERY_BOTH from its own rollouts and rewards.

Governing principle

Gemini counterfactual outcomes are a capability signal, not the weak model's routing oracle. The final policy must come from student rollouts plus reward, because what a strong model can solve directly says nothing about what the weak model should attempt directly.

This one principle shapes everything downstream: strict routing labels (6,517 steps) are used only as light exposure in SFT, never as the imitation target; the ablation grid exists precisely to prove the lift comes from RL-learned routing rather than SFT memorization.

Data & Splits

Clean split discipline

Overlap audit: 0 same-task / 0 same-step between skill pool and either training or benchmark splits.

Frozen skill pool

Deliberately not compressed further: 56.3% of clusters have a single clean member and ~80% have at most three, so further merging would hurt retrieval coverage while only deduplicating a few high-frequency GUI patterns.

Counterfactual Matrix

Oracle-train counterfactual matrix

Every oracle-train step was run under four conditions with Gemini 3.5 Flash: 41,116 unique steps × 4 = 164,464 condition rows (42 shards, 1 error total).

Step-level structure (what the numbers mean)

High-confidence routing labels total 6,517 (QUERY_TEXT 3,179 · QUERY_IMAGE 1,515 · QUERY_BOTH 1,196 · DIRECT_STRICT 627) — suitable for light exposure only.

Training Plan

MMSkill-RL = minimal SFT warm start + outcome-based RL

SFT (80–100K examples): format/schema ~5K · skill-conditioned action ~55K · direct action ~15–18K · light routing exposure ~6K · fallback reformats ~10–15K. Goal: legal structure, the four-way decision format, and skill-evidence-to-action generation — not routing imitation.

RL (full oracle-train): 41,116 prompts × 4 rollouts (164,464 trajectories/epoch), scaling to ×8. Weighted sampler instead of uniform: skill_rescue 3–4×, fallback 1.5–2×, direct 1×, uncovered 0.5×. Reward = format + deterministic action correctness + query cost (DIRECT < single query < BOTH) + skill-use reward only when the query improves the outcome.

Required ablations

The grid answers the reviewer question the project is built around: does the lift come from RL-learned skill use, or did SFT already teach the routing?

Run Status

Current runs and gates

Results

SFT vs base · SkillUseBench 4K (2026-08-13, Qwen3.5)

The red flag that matters most

Base routing spreads across DIRECT 3,111 / QUERY_TEXT 881; SFT routing collapses to QUERY_IMAGE on 4,000 of 4,000 steps, with query cost rising 0.011 → 0.080. The accuracy lift is real, but the policy learned "always fetch a visual exemplar," not when to fetch one.

This is the project's thesis confirming itself: supervised exposure teaches format and skill-conditioned action, and it does not teach routing. The routing decision has to be earned through student rollouts against outcome and cost rewards — exactly the RL stage that runs next. Discordant pairs: SFT-only-correct 2,209 vs base-only-correct 40.

Artifacts

Artifacts