Policy-Coupled Sensing, Memory, and Action

Overview

One thesis, one instrument

Thesis. An always-on personal AI is three policy-bearing functions, sensing, memory, and action, coupled by five auditable channels into one loop that closes through the world. Module presence is cheap; what is scarce is evidence that state crossing a channel causally changes a later decision.

Instrument. LoopAudit turns that claim into paired interventions: block or neutralize one channel's payload, hold everything else fixed, and watch whether downstream behavior changes. The P0 harness passed its first real-system audit (mem0) on 2026-08-20.

Proposal

Problem

Assistants are leaving the chat window for continuous interfaces: glasses, earbuds, browsers, operating systems. None of the adjacent literatures asks the question this creates: how is a task noticed in a continuous stream, and whether, when, and how should the system act at all? Each community ships one strong module and inherits silent assumptions about the others; the assumptions collide on a shared always-on interface.

Core hypotheses

1. The binding constraint is integration, not capability. Perception, memory, and tool use are each well served by foundation models; what fails is the coupling between them under privacy, budget, and interruption limits.
2. Coupling must be evidenced, not asserted. A camera, a vector store, and a planner can coexist while no payload demonstrably crosses between them. Channel-local intervention is the only attribution that separates a genuinely closed loop from better zero-shot perception.
3. Closure lags capability. Execution skill has raced to delegated action while personal-state closure typically stops at conditioned recall, and outcome-adaptive assistance is unoccupied.

Contributions (define → survey → operationalize)

1. A unifying systems framework — every surveyed work receives a checkable position: module, space-or-policy contribution, channels evidenced, binding regime.
2. A comprehensive framework-organized review — five communities, module spaces and policies, integrated systems on four deployment surfaces, and the benchmark landscape.
3. LoopAudit, an executable audit protocol — canonical trace, evidence levels E1–E4, five paired channel interventions, adapter and replay contracts, plus L0–L5 closure levels and a system card.

Idea Trajectory

How the framing evolved

2026-04 · ProEgoMem shelved → survey pivot — The systems paper was parked; its taxonomy and paper list seeded the survey corpus.

2026-04 · Landscape scan: the narrow gate — Deng/Liao's proactive-dialogue trilogy blocked a broad "proactive agents" survey. Interim conclusion: retreat to the Memory × Proactive × Egocentric intersection. superseded

2026-07 · Positioning reversal: the big umbrella — Confirmed the opposite bet: a system-and-applications survey entered through the wearable interface as forcing function, not a niche intersection.

2026-08-05 · Scoop crisis → three-axis differentiation — arXiv 2606.30306 "Always-On Agents" (136 pp) landed on the same phrase. Resolution after full-text autopsy: they audit the ledger (persistent-state lifecycle, legitimacy), we audit the loop (signal around sensing–memory–action, efficacy); their return arc closes through state governance, ours closes through the world; their typed event stream cannot express our two sensing-side channels.

2026-08-08 · Framing freeze — Rejected for v1: ten-policy top-level factorization, fast/slow dual-loop reserved for RSI position paper, action-as-world-model world models demoted to one anticipation mechanism, past/present/future as chapter structure demoted to interpretive layer.

2026-08-14 · The policy-coupled restructure — Channels renamed and given named coupling policies (Gating → Action Conditioning, Memory Prior → Sensing Prior, each with a κ-policy); modules became policy-bearing functions with a shared space/policy lens; LoopAudit became its own chapter; the old headline counts were withdrawn pending recoding — the former Gate column had mixed act-or-silence gating with memory-to-action conditioning.

2026-08-19 · L0–L5 closure levels — A driving-automation-style ladder anchored in the channel chain: a system's level is the longest closed prefix, judged by intervention evidence, with capability and proof burden rising together. Operating regimes were simultaneously narrowed from a global organizing axis to a diagnostic: same loop, different binding question. regimes-as-structure dropped

2026-08-20 · Claims vs. verdicts — The ingestion stage's channel annotations were demoted from audit findings to claims: unclaimed channels report "not claimed (absence not verified)", manipulation checks void mis-targeted interventions, and the deterministic engine signs only what it ran.

Experiments · LoopAudit

LoopAudit P0 · two-stage harness

Architecture. Agents onboard the system; a deterministic engine audits it. Any coding agent may serve as the ingestion stage (Codex produced the mem0 adapter); an acceptance gate (smoke run + manipulation check) freezes the adapter; paired runs, structured-diff verdicts, and closure levels contain no LLM.

Audit 1 · Toy assistant (planted defect)

The inert channel is the runtime version of Ding et al.'s "state written but never read is inert" — they can only state the principle; the harness measures it.

Audit 2 · mem0 (real system, offline-deterministic)

Closure level: L2, bottleneck timing — correct and self-consistent: a pure memory layer makes no initiation decisions, so L2 is its honest ceiling. Three engine bugs (verifier substring match, level-prefix logic, phrase-token regression) surfaced only when a real system was plugged in; none had appeared with the toy.

Run Status

Current runs and gates

Results

What the survey finds

Evidence is asymmetric around the loop. Local mechanisms are abundant; isolated evidence that one module changes another is sparse. Write and action-timing mechanisms are commonly described, while sensing-prior influence, post-action verification, and attributed policy update are rarely isolated. Executable environments and personal systems hold complementary halves of auditability: the former replay outcomes but start from explicit tasks, the latter own continuous context but rarely replay.

Closure lags capability. By reported evidence, products cluster at L1–L3: ChatGPT memory at conditioned recall, Windows Recall at persistent capture, Pulse at calibrated initiative. Computer-use agents combine L4 execution skill with L1–L2 personal-state closure, and the mismatch is the diagnosis. L5, outcome-adaptive assistance with attributed longitudinal benefit, is unoccupied in our corpus.

The wearable band shows the frontier precisely. Within-session memory measurably changes situated assistance; no surveyed system yet shows a dismissed or accepted intervention revising memory across weeks. That missing longitudinal edge, not any missing component, is the integration gap.

Paper Outline

Manuscript structure (11 chapters, 67 pp)

Open the live interactive outline ↗ *— the coauthor-facing structure view, kept in sync with every manuscript edit*

Every chapter closes with a takeaway box whose last line points at the audit column it leaves open.

Artifacts

Artifacts