Shuai Liu

Multimodal agent learning · 2026

MMSkill-RL

Can a visual agent learn when to act directly, invoke a text skill, use a visual skill, or combine both?

Proposal

Problem

Small VLMs powering GUI agents fail on the long tail of software interactions — the menu paths, dialog sequences, and app-specific conventions that larger models absorb parametrically. An external multimodal skill memory (procedural text cards + visual exemplars mined from demonstrations) can supply the missing knowledge, but consulting it is not free: evidence costs tokens and latency, and the useful modality varies per step. The learning problem is therefore not "can the model use a skill once given" but can a weak model learn when to consult which modality — DIRECT / QUERY_TEXT / QUERY_IMAGE / QUERY_BOTH — from its own outcomes.

Current work

Why significant

This is where the paper lives — five claims, each load-bearing:

  1. It isolates the decision, not the capability. The counterfactual matrix (41,116 steps × 4 conditions under a strong model) gives every training step a ground-truth need profile: 14.3% are rescued only by skills, 6.6% specifically need visual evidence, 43% need nothing, 42.7% are covered by nothing. Once these numbers exist, "when to consult" stops being an anecdote and becomes a measurable quantity — and every fixed policy is provably suboptimal at both ends (always-query burns cost on the 43%; never-query forfeits the 14.3%).
  2. It is the first clean testbed for modality-level routing. Evidence construction v2 makes modality the only variable: all skill conditions share the same top-2 packages, exemplar budget, and token budget. Any Text/Image/Both performance gap is attributable to modality utility rather than retrieval breadth — the confound that silently invalidates naive comparisons.
  3. It demonstrates the SFT ceiling empirically, not rhetorically. Our 2026-08-20 eval shows supervised exposure produces routing that is diverse but non-discriminative — the decision distribution is healthy yet uncorrelated with whether a step needs evidence. This is a reusable negative result: imitation can teach format and evidence-conditioned execution, but it cannot teach when, because the supervision itself cannot know what the student needs.
  4. RL closes exactly the gap that the analysis exposes. Outcome + query-cost reward over the student's own rollouts is the only signal that is about the student. If the RL stage lifts accuracy at equal or lower consultation cost — or matches accuracy at ≥30% lower cost — the paper establishes that when-to-consult is learnable from outcomes. The mechanism is general: deferral, tool-call gating, and retrieval triggering are the same decision wearing different clothes; GUI agents are the demonstration, not the boundary.
  5. Cost-aware autonomy is the deployment story. On-device and latency-budgeted agents cannot afford always-on retrieval; a policy that knows when not to look is what makes external memory practical at the edge.

Contribution targets

  1. A clean multimodal GUI skill pool — deduplicated, provenance-tracked, frozen.
  2. A skill-sensitive benchmark and oracle-training signal — every step labeled by which conditions (direct/text/image/both) a strong model can solve it under.
  3. Evidence that explicit skill use reduces none/uncovered and lifts weak-model action correctness.
  4. An end-to-end skill-use policy — the weak model learns DIRECT / QUERY_TEXT / QUERY_IMAGE / QUERY_BOTH from its own rollouts and rewards.

Governing principle

Gemini counterfactual outcomes are a capability signal, not the weak model's routing oracle. The final policy must come from student rollouts plus reward, because what a strong model can solve directly says nothing about what the weak model should attempt directly.

This one principle shapes everything downstream: strict routing labels (6,517 steps) are used only as light exposure in SFT, never as the imitation target; the ablation grid exists precisely to prove the lift comes from RL-learned routing rather than SFT memorization.

Data & Splits

Clean split discipline

split buckets scale role
skill-source 15–49 5,144 steps · 1,500 tasks skill pool provenance
oracle-train 50–99 41,116 steps · 2,493 tasks counterfactual matrix + RL prompts
external-final 0–14 12,564 steps · 761 tasks held-out benchmark only

Overlap audit: 0 same-task / 0 same-step between skill pool and either training or benchmark splits.

Evidence construction v2 (parity fix, 2026-08-06)

All skill conditions now share the same top-2 retrieved packages; only the view differs (text card / visual exemplars / both). Exemplar images are materialized at reduced resolution (~301k px, ~270 tokens each) while the current screenshot keeps full resolution for grounding. This removes the retrieval-breadth confound (Both previously saw up to 6 packages) and fixes the Both-overflow pathology: under the Qwen processor, Both sequences dropped from p50 = 27.5k tokens (100% over the old 9,216 cap — effectively never trained) to p50 = 4.5k, giving 0% truncation at cutoff 8,192 across all conditions.

Frozen skill pool
text clusters 3,010 (LLM-summary clustering over AgentNet raw skills)
visual exemplars 5,144 (≤3 clean exemplars per cluster)
embeddings 8,154

Deliberately not compressed further: 56.3% of clusters have a single clean member and ~80% have at most three, so further merging would hurt retrieval coverage while only deduplicating a few high-frequency GUI patterns.

Training Plan

MMSkill-RL = minimal SFT warm start + outcome-based RL

SFT (80–100K examples): format/schema ~5K · skill-conditioned action ~55K · direct action ~15–18K · light routing exposure ~6K · fallback reformats ~10–15K. Goal: legal structure, the four-way decision format, and skill-evidence-to-action generation — not routing imitation.

RL (full oracle-train): 41,116 prompts × 4 rollouts (164,464 trajectories/epoch), scaling to ×8. Weighted sampler instead of uniform: skill_rescue 3–4×, fallback 1.5–2×, direct 1×, uncovered 0.5×. Reward = format + deterministic action correctness + query cost (DIRECT < single query < BOTH) + skill-use reward only when the query improves the outcome.

Required ablations

arm answers
RL-only can rewards alone train format and routing?
minimal SFT + RL main method
full SFT only is SFT enough? (the 8-13 result already says no for routing)
full SFT + RL does heavy imitation help or hurt RL?

The grid answers the reviewer question the project is built around: does the lift come from RL-learned skill use, or did SFT already teach the routing?