The Intent Graph: Representing User Intent Across Modalities

The Intent Graph

When a surgeon glances at a monitor, reaches for an instrument, and mutters a half-formed instruction — they are expressing a single intent through three channels simultaneously. Current multimodal systems process each channel independently and then vote. We argue this is wrong.

The problem with independent processing: it discards the temporal binding between modalities. The gaze fixation that precedes the gesture by 80ms is not coincidental — it's the anticipatory signal that makes the gesture interpretable.

The Graph Structure

We represent the ongoing interaction state as a directed graph where:

Intent resolution is a graph classification task: given the current subgraph window (typically 200–400ms), predict the most probable intent from a vocabulary of ~2,000 learned intent primitives.

Latency Breakdown