Our Projects

GTA ▾

Method Results Examples Citation

GGVL

ECCV 2026

From Gaze to Meaning: A Training-Free
AI Agent for Unified Grounding and Explanation

Shayan Nasiriboukani¹, Sara Atito^1,2*, Mohammad Nezamipour¹, Muhammad Awais^1,2*

¹Centre for Vision, Speech and Signal Processing, University of Surrey, UK

²Institute for People-Centered AI, University of Surrey, UK

^* Corresponding authors

arXiv Code

These GTA game videos are included only as a fun visual demonstration of gaze annotation. They are not used in the paper, training, evaluation, datasets, or reported results.

Abstract

Understanding human attention is fundamental for scene interpretation, yet existing approaches often rely on heavily trained models that lack interpretability. Prior methods struggle to jointly reason about gaze targets, attended objects, and visual grounding without extensive supervision.

To the best of our knowledge, this work introduces the first training-free Gaze Target Agent (GTA) for gaze-guided reasoning across tasks such as gaze target prediction, attention localization, and object identification. This is achieved by leveraging pretrained vision language models, augmenting them with visually guided prompts, and employing a memory-based retrieval strategy for high-uncertainty samples to improve performance without additional training.

We evaluate our approach using both quantitative metrics and qualitative results. Quantitatively, our method achieves state of the art performance on the GazeFollow and GazeHOI benchmarks. Qualitatively, our agent provides detailed semantic predictions, predicts the correct targets even when ground truth labels are wrong, and remains flexible without vocabulary constraints.

Method

Architecture Overview

GTA is a fully training-free agent that chains three pretrained components: (1) a gaze estimator that converts each person's line of sight into a visual arrow overlay, (2) a vision-language model that reads these overlays to name the attended object, and (3) an uncertainty-aware RAG module that retrieves similar training examples from memory whenever the VLM is unsure, sharpening predictions without updating any parameters. A dedicated grounding stage then localises the predicted object in the image using spatial overlap with the predicted gaze heatmap.

Experiments

State-of-the-Art Comparison

GTA outperforms prior method on both benchmarks with zero learnable parameters.

GazeFollow: Post-Camera-Ready Evaluation

Method	Params	Input	Acc@1 ↑	Acc@3 ↑	MultiAcc@1 ↑
Tafasca et al.	116M	I+V	0.447	0.642	0.516
Qwen3_{[2B (GGVL)]}	0	I	0.450	0.684	0.506
Qwen3_{[2B (GTA)]}	0	I	0.499	0.761	0.561

Table 1. Params denotes the number of learnable parameters. This evaluation was carried out after the ECCV camera-ready deadline: we found that giving the model only the labels of retrieved examples, rather than the retrieved images themselves, gives better results.

GazeHOI: Post-Camera-Ready Evaluation

Method	GazeAcc ↑	Acc@1 ↑	Acc@3 ↑	AP@50 ↑	mIoU ↑
Tafasca et al.^†	0.652	0.306	0.463	--	--
Tafasca et al.	0.723	0.583	0.706	--	--
Qwen3_{[2B (GGVL)]}	0.731	0.517	0.721	0.192	0.408
Qwen3_{[2B (GTA)]}	0.731	0.544	0.758	0.192	0.408

Table 2. The † symbol indicates the previous state-of-the-art training-free result. This evaluation was carried out after the ECCV camera-ready deadline: we found that giving the model only the labels of retrieved examples, rather than the retrieved images themselves, gives better results.

GazeFollow: Camera-Ready Result

Method	Params	Input	Acc@1 ↑	Acc@3 ↑	MultiAcc@1 ↑
Tafasca et al.	116M	I+V	0.447	0.642	0.516
Qwen3_{[2B (GGVL)]}	0	I	0.450	0.684	0.506
Qwen3_{[2B (GTA)]}	0	I	0.493	0.728	0.558

Table 3. Params denotes the number of learnable parameters.

GazeHOI: Camera-Ready Result

Method	GazeAcc ↑	Acc@1 ↑	Acc@3 ↑	AP@50 ↑	mIoU ↑
Tafasca et al.^†	0.652	0.306	0.463	--	--
Tafasca et al.	0.723	0.583	0.706	--	--
Qwen3_{[2B (GGVL)]}	0.731	0.517	0.721	0.192	0.408
Qwen3_{[2B (GTA)]}	0.731	0.545	0.745	0.192	0.408

Table 4. The † symbol indicates the previous state-of-the-art training-free result.

Ablation Across VLM Backbones

Method	GazeHOI		GazeFollow
Method	Acc@1 ↑	Acc@3 ↑	Acc@1 ↑	Acc@3 ↑	MultiAcc@1 ↑
Public
Qwen3_{[2b (GGVL)]}	0.517	0.721	0.450	0.684	0.506
Qwen3_{[2b (GTA)]}	0.545	0.745	0.488	0.717	0.550
Llava 1.5_{[4b (GGVL)]}	0.528	0.691	0.380	0.630	0.437
Llava 1.5_{[4b (GTA)]}	0.572	0.750	0.466	0.700	0.535
Qwen3_{[4b (GGVL)]}	0.516	0.742	0.543	0.722	0.613
Qwen3_{[4b (GTA)]}	0.564	0.758	0.559	0.769	0.636
Llava_{[8b (GGVL)]}	0.548	0.718	0.409	0.676	0.468
Llava_{[8b (GTA)]}	0.557	0.771	0.484	0.726	0.553
Qwen3_{[8b (GGVL)]}	0.533	0.742	0.584	0.755	0.650
Qwen3_{[8b (GTA)]}	0.602	0.772	0.602	0.778	0.668
Private
Gemini 2.5 Flash_[GGVL]	0.617	0.741	0.594	0.773	0.655
Gemini 2.5 Flash_[GTA]	0.621	0.753	0.622	0.789	0.679
Gemini 2.5 pro_[GGVL]	0.625	0.728	0.584	0.767	0.649
Gemini 2.5 pro_[GTA]	0.631	0.739	0.608	0.787	0.665

Table 5. Results are reported under identical settings to show the effect of reasoning across model scales and architectures.

Uncertainty Threshold Ablation

Threshold	Acc@1 ↑	Acc@3 ↑	MultiAcc@1 ↑
0	42.64	71.25	49.23
15	47.85	71.77	54.33
25	48.49	71.83	54.77
35	48.47	71.54	54.75
45	47.76	70.85	53.97
55	47.09	70.03	53.09
Dynamic	48.80	71.70	55.50

Table 6. We compare fixed thresholds with the dynamic uncertainty threshold used in GTA.

Visual Prompt Ablation

Prompt	Acc@1 ↑	Acc@3 ↑	MultiAcc@1 ↑
Head bbox	28.69	67.57	34.36
Blue dot	40.99	66.90	47.49
Gray scale	38.39	69.72	44.42
Blur	38.98	67.75	44.92
Red arrow	48.29	72.08	54.45
Green arrow	43.91	71.98	50.17
Yellow arrow	48.06	72.21	54.25
Blue arrow	48.80	71.70	55.50
3px blue arrow	48.41	72.25	54.83
5px blue arrow	49.29	72.81	55.79
6px blue arrow	49.10	72.84	55.40

Table 7. We compare different prompt types, arrow colors, and arrow widths. The color arrows without a stated width use 4px.

Retrieval-Size Ablation

k	Acc@1 ↑	Acc@3 ↑	MultiAcc@1 ↑
3	48.68	72.75	55.02
4	48.56	72.69	54.89
5	49.21	72.81	55.58
6	49.29	72.81	55.79
10	49.04	72.27	55.37

Table 8. We vary the number of retrieved examples used during retrieval-based refinement.

Vocabulary-Setting Ablation

Vocabulary	Acc@1 ↑	Acc@3 ↑	MultiAcc@1 ↑	E5 Acc@1 ↑	E5 Acc@3 ↑
Fixed + GGVL	45.75	69.78	51.48	48.43	72.40
Fixed + GTA	49.29	72.81	55.79	51.48	76.29
Open + GGVL	30.84	43.75	35.59	40.11	57.28
Open + GTA	40.63	54.27	46.07	47.85	64.62

Table 9. We compare fixed and open-vocabulary evaluation for GGVL and GTA using both exact matching and E5 semantic matching.

1 / 9

Qualitative Evaluation

What GTA Can Do

Vocabulary-Free Flexibility

Without any vocabulary constraints, GTA generates expressive open-ended predictions that adapt naturally to any scene.

Correct Even When Labels Are Wrong

Our agent identifies the true gaze target even in cases where ground-truth annotations are inaccurate or ambiguous.

Richer Semantic Predictions

GTA produces detailed, context-aware descriptions of attended objects, going beyond single-word category labels.

1 / 3

Citation

Cite This Work

If you find GTA useful in your research, please cite our paper:

@inproceedings{nasiriboukani2026gta,
  author    = {Nasiriboukani, Shayan and Atito, Sara and
               Nezamipour, Mohammad and Awais, Muhammad},
  title     = {From Gaze to Meaning: A Training-Free AI Agent
               for Unified Grounding and Explanation},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026},
}

From Gaze to Meaning: A Training-Free AI Agent for Unified Grounding and Explanation