Affogato Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale

Abstract

Affordance grounding, localizing object regions given natural language interaction descriptions, is fundamental for embodied agents yet remains bottlenecked by the scarcity of large-scale training data. Manual affordance annotation is prohibitively expensive, confining existing datasets to narrow object and affordance categories and preventing open-vocabulary generalization. We introduce Affogato, a benchmark for open-vocabulary affordance grounding. At its core is Affogato-750K, a large-scale dataset of 750K open-vocabulary 3D affordance annotations, constructed through an automated pipeline that leverages foundation models for affordance query generation, interaction-point prediction, and mask generation, followed by multi-view aggregation. It covers significantly more diverse categories than any existing dataset, with 5K human-verified test pairs ensuring evaluation reliability. We further present Espresso-3D and Espresso-2D, simple yet effective baselines sharing the same architecture across 3D and 2D. Pretraining on Affogato-750K further boosts both Espresso and existing methods, yielding substantial gains especially on unseen object and affordance categories, demonstrating the dataset's effectiveness as a scalable source of open-vocabulary affordance supervision.

Data Annotation Pipeline

Our data annotation pipeline automatically generates high-quality affordance annotations by chaining three foundation models. Given multi-view renderings of a 3D object, Gemma3 generates open-vocabulary natural language affordance queries via chain-of-thought prompting, Molmo points to the interaction region for each query, and MobileSAM converts each point into a 2D segmentation mask. The multi-view masks are then lifted and aggregated on the 3D object surface via cross-view voting to obtain a final 3D affordance heatmap.

Espresso Model Architecture

We present Espresso, a unified baseline designed to cleanly validate the contribution of Affogato-750K. Building on a shared architecture, we instantiate two models: Espresso-3D for 3D point clouds and Espresso-2D for 2D images. Each consists of a modality-specific visual encoder, a text encoder, and a text-conditioned heatmap decoder. The core of our design is a heatmap decoder that replaces learnable queries with text embeddings, naturally supporting open-vocabulary affordance grounding without predefined categories. The affordance heatmap is predicted as the cross-modal similarity between the vision representation and the conditioned text embedding.

Annotation Quality

To construct a reliable benchmark, we conducted a human evaluation on 5K randomly sampled affordance query–heatmap pairs. The evaluation shows strong agreement between the automatically generated annotations and human judgment, achieving an 84.8% success rate. This human validation process is also used to construct the test split: pairs with incorrect queries are removed, and pairs with incorrect interaction points are manually corrected, yielding a reliable, high-quality evaluation benchmark.

Zero-shot Transfer to Robot Manipulation

As a harder test of real-world transfer, we evaluate Espresso-2D zero-shot on the Open X-Embodiment dataset, whose robot-manipulation scenes are visually distant from any Objaverse render and were never seen during pretraining. Despite this large domain gap, Espresso-2D still localizes the queried affordance accurately, demonstrating that Affogato-750K's synthetic-only supervision transfers to real robot inputs.

“Grasp the banana”

“Grasp the knife”

“Open the box drawer”

“Open the oven”

Citation

@article{lee2025affogato,
  title={Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale},
  author={Lee, Junha and Park, Eunha and Park, Chunghyun and Kang, Dahyun and Cho, Minsu},
  journal={arXiv preprint arXiv:2506.12009},
  year={2025}
}