AffoGato: Learning Open-Vocabulary Affordance Grounding with Foundation Models

Abstract

Affordance grounding — localizing object regions based on natural language descriptions of interactions — is a critical challenge for enabling intelligent agents to understand and interact with their environments. However, this task remains challenging due to the need for fine-grained localization, the ambiguity arising from multiple valid interaction regions, and the scarcity of large-scale datasets. We introduce AffoGato, a unified framework for open-vocabulary affordance grounding across both 3D and 2D. Our approach leverages supervision from foundation models to automatically generate scalable affordance annotations, enabling training without reliance on exhaustive manual labeling. As part of this pipeline, we construct Affo-150K, a large-scale automatically generated dataset of 150K 3D object instances with free-form affordance descriptions and corresponding 3D affordance heatmaps. Within AffoGato, we design simple yet effective models, Gato-3D and Gato-2D, by combining pre-trained part-aware vision encoders with text-conditional heatmap decoders. Our models achieve state-of-the-art performance across existing 3D and 2D benchmarks, with pretraining on Affo-150K further enhancing their open-vocabulary capabilities.

Data Annotation Pipeline

Our data annotation pipeline automatically generates high-quality affordance annotations by combining three foundation models. Given multi-view renderings of a 3D object, Gemma3 generates natural language affordance queries, Molmo points to the interaction regions, and MobileSAM produces precise segmentation masks. The multi-view mask logits are then aggregated on the 3D object surface to obtain affordance heatmaps.

The Affo-150K Dataset

Affo-150K is the largest and most diverse dataset for affordance grounding to date, comprising 150K 3D object instances from Objaverse across four categories: Daily-Used, Furnitures, Transportations, and Electronics. Each object includes approximately 5 affordance query-heatmap pairs, totaling 750K annotations. Unlike existing datasets constrained by predefined taxonomies, Affo-150K achieves truly open-vocabulary coverage with >450 object classes and >350 affordance types.

Dataset	Cov. ↑	Div. ↑
LASO (2024)	0.6384	0.6578
Affo-150K	0.7532	2.6638

Dataset Comparison

Gato Model Architecture

We present a minimalistic architecture for affordance grounding, dubbed Gato. Building on a shared architectural concept, we create two models: Gato-3D for 3D point clouds and Gato-2D for 2D images. Each model consists of a modality-specific visual encoder, a text encoder (CLIP), and a text-conditioned heatmap decoder. The affordance heatmap is predicted as the cross-modal similarity of the vision representation and the conditioned text embedding.

Results

3D Affordance Grounding

Gato-3D achieves state-of-the-art performance on the LASO benchmark for both seen and unseen settings. When pretrained on Affo-150K (Gato-3D*), the model shows significant improvements, particularly on unseen object categories — demonstrating the value of large-scale pretraining for generalization.

2D Affordance Grounding

Gato-2D demonstrates exceptional zero-shot generalization capability on the AGD20K benchmark. Despite being a lightweight combination of DINOv2 and CLIP, our model significantly outperforms heavily parameterized LLM-based approaches. When pretrained on Affo-150K and finetuned with full supervision (Gato-2D*), the model achieves the best performance across all metrics.

Annotation Quality

We conducted a systematic quality assessment on 5K randomly sampled affordance query-heatmap pairs. The evaluation demonstrated an 84.8% pass rate, confirming the robustness of our automatic annotation pipeline. The analysis also reveals common failure modes across the three pipeline stages.

Citation

@article{lee2025affogato,
  title={Affogato: Learning Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale},
  author={Lee, Junha and Park, Eunha and Park, Chunghyun and Kang, Dahyun and Cho, Minsu},
  journal={arXiv preprint arXiv:2506.12009},
  year={2025}
}