Computational Models of Affordances in Robotics.

Abstract: J. J. Gibson’s concept of affordance, one of the central pillars of ecological psychology, is a truly remarkable idea that provides a concise theory of animal perception predicated on environmental interaction. It is thus not surprising that this idea has also found its way into robotics research as one of the underlying theories for action perception. The success of the theory in this regard has meant that existing research is both abundant and diffuse by virtue of the pursuit of multiple different paths and techniques with the common goal of enabling robots to learn, perceive, and act upon affordances. Up until now, there has existed no systematic investigation of existing work in this field. Motivated by this circumstance, in this article, we begin by defining a taxonomy for computational models of affordances rooted in a comprehensive analysis of the most prominent theoretical ideas of import in the field. Subsequently, after performing a systematic literature review, we provide a classification of existing research within our proposed taxonomy. Finally, by both quantitatively and qualitatively assessing the data resulting from the classification process, we highlight gaps in the research terrain and outline open questions for the investigation of affordances in robotics that we believe will help inform future work, prioritize research goals, and potentially advance the field toward greater robot autonomy.

Original publication: Sage (+errata)

Authors Copy: Link

Download the data as MS Excel file (50 KB): Download»

Title	Authors	Year	Perception Perspective	Level	Order	Temporality	Selective Attention	Abstraction	Competitive	Chaining	Acquisition	Prediction	Generalization	Exploitation	Learning	Kind	Abstraction	Brain Areas	Method	Features	Training	Evaluation	Data added
Title	Authors	Year	Perception Perspective	Level	Order	Temporality	Selective Attention	Abstraction	Competitive	Chaining	Acquisition	Prediction	Generalization	Exploitation	Learning	Kind	Abstraction	Brain Areas	Method	Features	Training	Evaluation	Data added
A model-based approach to finding substitute tools in 3D vision data	Abelha et al.	2016	agent	local	0th	stable	no	micro	yes	no	ground truth	Optimization	yes	action selection	offline	tool-use	mathematical		geometric part fitting	point clouds, superquadrics	supervised	benchmark	original study
Unsupervised learning of affordance relations on a humanoid robot	Akgun et al.	2009	agent	meso	1st	stable	no	micro	no	no	exploration	Classification	yes	action selection	offline	rollability	mathematical	not specified	SOM, SVM	shape, size	unsupervised	real robot	original study
Supervised learning of hidden and non-hidden 0-order affordances and detection in real scenes	Aldoma et al.	2012	agent	global	1st	stable	no	micro	no	no	ground truth	Classification	yes	not specified	offline	general	mathematical	not specified	SVM, Boost, RF	SEE, SHOT, NDS, SI, PFH	supervised	benchmark	original study
From Human Instructions to Robot Actions: Formulation of Goals, Affordances and Probabilistic Planning	Antunes et al.	2016	agent	global	2nd	stable+variable	no	micro+macro	yes	yes	exploration	Inference	yes	planning	online	pulling, dragging, grasping	mathematical		BN, PRAXICON Semantic Network, PRADA Planner	2D geom feat., 2D tracked object displacement	not specified	real robot	original study
On Exploiting Haptic Cues for Self-Supervised Learning of Depth-Based Robot Navigation Affordances	Baleia et al	2015	agent	global	1st	stable	no	micro	no	no	exploration	Inference	yes	planning	online	traversability	mathematical		histograms, clustering, similarity metrics	depth, haptic	self-supervised	real robot	original study
Self-supervised learning of depth-based navigation affordances from haptic cues	Baleia et al.	2014	agent	global	1st	stable	no	micro	no	no	exploration	Inference	yes	planning	online	traversability	mathematical		histograms, clustering, similarity metrics	depth, haptic	self-supervised	real robot	original study
Learning grasping affordance using probabilistic and ontological approaches	Barck-Horst et al	2009	agent	meso	1st	stable	no	micro	no	no	exploration	Inference	yes	planning	online	grasping	mathematical		voting function, ontological rule-engine	shape, size, grasp region, force	self-supervised	simulation	original study
Grasp affordances from multi-fingered tactile exploration using dynamic potential fields	Bierbaum et al.	2009	agent	local	1st	stable	no	micro	no	no	exploration	Regression	yes	planning	offline	grasping	mathematical		Potential Fields	planar faces of object	self-supervised	simulation	original study
Behavioural plasticity in evolving robots	Carvalho and Nolfi	2016	agent	global	1st	stable	no	micro	no	no	exploration	Regression	yes	action selection	online	traversability	mathematical		NN	depth, haptic	self-supervised	simulation	original study
Using Object Affordances to Improve Object Recognition	Castellini et al.	2011	agent	meso	1st	stable	no	micro	no	no	demonstration	Regression	yes	action selection	offline	grasping	mathematical		MLP, SVM, k-means, histograms	SIFT BoW, contact joints	supervised	benchmark	original study
A Probabilistic Concept Web on a Humanoid Robot	Celikkanat at al.	2015	agent	meso	1st	stable	no	micro	no	no	exploration	Optimization	no	language	offline	pushing, grasping, throwing, shaking	mathematical		MRF, Loopy belief propagation	depth, haptic, proprioceptive and audio	semi-supervised	real robot	original study
Determining proper grasp configurations for handovers through observation of object movement patterns and inter-object interactions during usage	Chan et al.	2014	agent	meso	1st	stable	no	micro	yes	no	demonstration	Optimization	yes	action selection	online	grasping	mathematical		k-means, nearest neighbor	pose, action-object relation	unsupervised	real robot	original study
A Bio-Inspired Robot with Visual Perception of Affordances	Chang	2015	agent	meso	1st	stable	no	micro	no	no	ground truth	Classification	yes	action selection	offline	cutting, painting	neural	AL, MB	ANN	edges, TSSC	supervised	real robot	original study
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving	Chen et al.	2015	agent	global	1st	stable+variable	no	micro	no	no	ground truth	Regression	yes	planning	offline	traversability	mathematical		CNN	RGB images, motor controls	supervised	simulation	original study
Learning haptic affordances from demonstration and human-guided exploration	Chu et al.	2016	agent	meso	1st	stable	no	micro	no	no	demonstration+exploration	Classification	yes	action selection	offline	openable, scoopable	mathematical		HMM	forces and torques	supervised	real robot	original study
Learning Object Affordances by Leveraging the Combination of Human-Guidance and Self-Exploration	Chu et al.	2016	observer	meso	1st	stable	no	micro	no	no	demonstration+exploration	Optimization	no	action selection	offline	pushing, openning, turning	mathematical		HMM	color, size, pose, force torque, robot arm pose	self-supervised	real robot	original study
Learning Affordances of Consummatory Behaviors: Motivation-Driven Adaptive Perception	Cos et al.	2010	agent	meso	1st	stable	no	micro	no	no	exploration	Classification	yes	planning	online	general	mathematical		GWRN+RL	illumination	supervised	simulation	original study
Training Agents with Interactive Reinforcement Learning and Contextual Affordances	Cruz et al.	2016	agent	meso	1st	stable+variable	no	micro	no	no	ground truth	Classification	no	action selection	offline	manipulation, locomotion	mathematical		DMLP	agent state, action, object	unsupervised	simulation	original study
Interactive reinforcement learning through speech guidance in a domestic scenario	Cruz et al.	2015	agent	meso	1st	stable	no	micro	no	no	ground truth	Classification	no	action selection	offline	graspable, dropable, moveable, cleanable	mathematical		DMLP	robot state, intended action, object information	supervised	simulation	original study
A Cognitive Control Architecture for the Perception–Action Cycle in Robots and Agents	Cutsuridis and Taylor	2013	agent	meso	1st	stable	yes	micro	yes	no	exploration	Inference	yes	action selection	online	grasping	neural	AIP, DVv	ANN	shape	unsupervised	real robot	original study
Learning Affordances for Categorizing Objects and Their Properties	Dag et al	2010	observer	meso	1st	stable	no	micro	no	no	demonstration	Classification	no	single-/multi-step prediction	offline	manipulation	mathematical	-	SVM, k-means, spectral clustering	3D position, orientation, shape, size	unsupervised	benchmark	original study
Semantic grasping: planning task-specific stable robotic grasps	Dang and Allen	2014	agent	local	1st	stable	no	micro	yes	no	exploration	Optimization	yes	action selection	offline	grasping	mathematical		Nearest Neighbor	grasp, shape context	supervised	real robot	original study
Denoising Auto-encoders for Learning of Objects and Tools Affordances in Continuous Space	Dehban et al.	2016	agent	meso	1st	stable	no	micro	no	no	exploration	Inference	yes	action selection	online	pulling, dragging	mathematical		Denoising auto-encoders	2D shape, object displacement	unsupervised	simulation+real robot	original study
Predicting Functional Regions on Objects	Desai and Ramanan	2013	agent	global	0th	stable	no	micro	no	no	ground truth	Optimization	yes	not specified	offline	grasping, support	mathematical		Deformable Part Models	HOG	supervised	benchmark	original study
Learning object-specific grasp affordance densities	Detry et al.	2009	agent	meso	1st	stable	no	micro	no	no	exploration	Optimization	no	planning	online	grasping	mathematical	not specified	KDE, Sampling	ECV	self-supervised	real robot	original study
Learning grasp affordance densities	Detry et al.	2011	agent	meso	1st	stable	no	micro	no	no	exploration	Optimization	no	planning	online	grasping	mathematical	not specified	KDE, Sampling	ECV	self-supervised	real robot	original study
Refining Grasp Affordance Models by Experience	Detry et al.	2010	agent	meso	1st	stable	no	micro	no	no	exploration	Optimization	no	action selection	online	grasping	mathematical		KDE, Sampling	ECV	self-supervised	real robot	original study
From primitive behaviors to goal-directed behavior using affordances	Dogar et al.	2007	agent	meso	2nd	stable	yes	micro	yes	yes	exploration	Classification	yes	action selection	offline	traversability	mathematical	not specified	k-Means, SVM	shape, distance	unsupervised	real robot	original study
Ecological Robotics	Duchon et al.	1998	agent	global	1st	variable	no	micro	no	no	exploration	Optimization	yes	planning	online	locomotion, survival	mathematical		law of control	optical flow	not specified	simulation+real robot	original study
Predicting the Intention of Human Activities for Real-Time Human-Robot Interaction (HRI)	Dutta and Zielinski	2016	agent	meso	1st	stable	no	micro	yes	no	ground truth	Optimization	yes	action selection	offline	reachable, pourable, movable, drinkable	mathematical		Heat maps	angular, location + dist. to object, sematic labels	supervised	simulation	original study
Discrete fuzzy grasp affordance for robotic manipulators	Eizicovits et al.	2012	agent	meso	1st	stable	no	micro	yes	no	demonstration	Optimization	no	planning	offline	grasping	mathematical		Affordance manifolds	wrist location, roll angle	supervised	real robot	original study
Learning structural affordances through self-exploration	Erdemir et al.	2012	agent	global	1st	stable	no	micro	no	no	exploration	Classification	yes	planning	offline	crawling	mathematical		SOM, k-means, LVQ	fixation point, motor values	semi-supervised	real robot	original study
A robot rehearses internally and learns an affordance relation	Erdemir et al.	2008	agent	meso	1st	stable	no	micro	no	no	exploration	Regression	no	planning	offline	traversabilty	mathematical		GMM	object edges	self-supervised	simulation+real robot	original study
Learning probabilistic discriminative models of grasp affordances under limited supervision	Erkan et al.	2010	agent	meso	1st	stable	no	micro	yes	no	ground truth	Optimization	yes	action selection	online	grasping	mathematical		Kernel Logistic Regression	ECV	semi-supervised	real robot	original study
Bootstrapping Relational Affordances of Object Pairs using Transfer	Fichtl et al.	2016	agent	meso	1st	stable	no	micro	yes	no	exploration	Classification	yes	single-/multi-step prediction	online	rake, pull/sh, move, lift, take, pour, slide	mathematical		Random Forests	pose, size; relational hist.feat./PCA on PCL	semi-supervised	simulation	original study
Learning About Objects Through Action - Initial Steps Towards Artificial Cognition	Fitzpatrick et al.	2003	agent	meso	1st	stable	no	micro	no	no	exploration	Optimization	no	action selection	offline	general	neural	F5/AIP	Histogram	shape, identity	unsupervised	real robot	original study
Neural Model for the Visual Recognition of Goal-Directed Movements	Fleischer et al.	2008	agent	meso	1st	stable	no	micro	yes	no	ground truth	Classification	yes	action selection	offline	grasping	neural	AIP	NN, k-means	orientation, object + hand shape, saliency of feat.	unsupervised	benchmark	original study
Learning Predictive Features in Affordance based Robotic Perception Systems	Fritz et al.	2006	agent	meso	1st	stable	no	micro	no	no	ground truth	Optimization	yes	action selection	offline	lifting	mathematical		k-means, MAP, decision tree	SIFT, color, mass-center, shape descr., actuator	supervised	simulation	original study
Visual Learning of Affordance Based Cues	Fritz et al.	2006	agent	meso	1st	stable	no	micro	no	no	ground truth	Classification	no	action selection	offline	lifting	mathematical		Nearest Neighbor, C4.5 Decision tree	SIFT	self-supervised	simulation	original study
Synergy-based affordance learning for robotic grasping	Geng et al.	2013	agent	meso	2nd	stable	no	micro	yes	no	demonstration+exploration	Classification	yes	planning	offline	grasping	neural	VIP, CIPS, 7a, 7b, AIP	Growing Neural Gas (GNG)	not specified	unsupervised	real robot	original study
Object recognition using visuo-affordance maps	Gijsberts et al.	2010	agent	meso	1st	stable	no	micro	no	no	ground truth	Regression	yes	single-/multi-step prediction	offline	grasping	mathematical		Regularized Least Squares	SIFT	supervised	benchmark	original study
Towards Lifelong Affordance Learning using a Distributed Markov Model	Glover and Wyeth	2016	agent	meso	1st	stable	no	micro	no	no	exploration	Inference	yes	action selection	online	grasping	mathematical		Distributed Markov Model	object pose, tactile readings	unsupervised	real robot	original study
Learning visual affordances of objects and tools through autonomous robot exploration	Gonçalves et al.	2014	agent	meso	1st	stable	no	micro	no	no	exploration	Inference	yes	action selection	offline	pulling, dragging	mathematical		BN	2D geom. feat., 2D tracked object displacement	self-supervised	simulation	original study
Learning intermediate object affordances: Towards the development of a tool concept	Gonçalves et al.	2014	agent	meso	1st	stable	no	micro	no	no	exploration	Inference	yes	action selection	offline	pulling, dragging	mathematical		BN, PCA, BN structure learning	2D geom. feat., 2D tracked object displacement	self-supervised	simulation+real robot	original study
A Behavior-Grounded Approach to Forming Object Categories: Separating Containers From Noncontainers	Griffitth et al.	2012	agent	meso	1st	stable	no	micro	no	no	exploration	Classification	yes	action selection	offline	drop, move, grasping, /shake	mathematical		SOM, Spectral clustering, PCA and k-NN	auditory and visual feature trajectories, depth	unsupervised	real robot	original study
Affordance in Autonomous Robot	Hakura et al.	1996	agent	meso	1st	stable	no	micro	no	no	exploration	Classification	yes	action selection	online	traversability	mathematical		ART, RL	pulse sensor readings	semi-supervised	simulation	original study
Intrinsically Motivated Affordance Discovery and Modeling	Hart and Grupen	2013	agent	meso	1st	stable	no	micro	yes	no	exploration	Optimization	yes	action selection	online	grasping	mathematical		RL	hue, shape, pose of object	unsupervised	real robot	original study
Attribute Based Affordance Detection from Human-Object Interaction Images	Hassan and Dharmaratne	2016	observer	meso	1st	stable	no	micro	no	no	demonstration	Classification	yes	action selection	offline	general	mathematical		BN, SVM	SIFT, HOG, textons, color hist., object attributes	supervised	benchmark	original study
Affordance-Based Grasp Planning for Anthropomorphic Hands from Human Demonstration	Hendrich and Bernardino	2014	agent	meso	1st	stable	no	micro	no	no	demonstration	Regression	yes	action selection	offline	grasping	mathematical		PCA, FK, IK	shape, size	supervised	real robot	original study
Decoupling behavior, perception, and control for autonomous learning of affordances	Hermans et al.	2013	agent	meso	1st	stable	no	micro	no	no	exploration	Regression	yes	action selection	online	pushing, pulling	mathematical		Feedback control	pose, depth	supervised	real robot	original study
Learning Contact Locations for Pushing and Orienting Unknown Objects	Hermans et al.	2013	agent	meso	1st	stable	no	micro	yes	no	exploration	Regression	yes	action selection	offline	pushing	mathematical		SVR	Histogram of points (in pose space)	semi-supervised	real robot	original study
Hallucinated Humans as the Hidden Context for Labeling 3D Scenes	Jiang et al.	2013	agent	meso	1st	stable	no	micro	no	no	demonstration	Optimization	yes	not specified	offline	general	mathematical		Infinite Factory Topic Models (DPPM)	human pose, object pose	supervised	benchmark	original study
Extracting whole-body affordances from multimodal exploration	Kaiser et al.	2014	agent	global	1st	stable	no	micro	no	no	hardcoded	Inference	yes	action selection	offline	support, lean, grasping, hold	mathematical		Reasoning	surface characteristics	supervised	real robot	original study
Validation of Whole-Body Loco-Manipulation Affordances for Pushability and Liftability	Kaiser et al.	2015	agent	global	2nd	stable	no	micro	no	no	hardcoded	Inference	no	action selection	not specified	pushing, lifting	mathematical		RANSAC, clustering	surface normals, area	not specified	real robot	original study
Representation and extraction of image feature associated with maneuvering affordance	Kamejima	2002	agent	global	1st	stable	no	micro	no	no	exploration	Regression	yes	planning	online	maneuverability	mathematical		Directional Fourier Imaging, Self similarity	scene image	unsupervised	real robot	original study
Anticipative generation and in-situ adaptation of maneuvering affordance in naturally complex scene	Kamejima et al.	2008	agent	global	1st	stable+variable	no	micro	no	no	exploration	Classification	yes	planning	online	maneuverability	mathematical		Fractal Coding	scene image	unsupervised	real robot	original study
Perceiving, learning, and exploiting object affordances for autonomous pile manipulation	Katz et al.	2014	agent	local	0th	stable	no	macro	yes	yes	ground truth	Classification	yes	action selection	offline	pushing, pulling, grasping	mathematical		SVM, PCA, Mean shift	PCA axes, size, center of gravity	supervised	real robot	original study
Semantic Labeling of 3D Point Clouds with Object Affordance for Robot Manipulation	Kim et al.	2014	agent	local	0th	stable	no	micro	no	no	ground truth	Regression	yes	not specified	offline	pushing, lifting, grasping	mathematical		Logistic regression, k-means	geometric features	supervised	benchmark	original study
Interactive Affordance Map Building for a Robotic Task	Kim et al.	2015	agent	local	2nd	variable	no	micro	no	no	ground truth	Optimization	yes	single-/multi-step prediction	offline	pushing	mathematical		Logistic regression, MRF	geometric features	supervised	simulation	original study
Traversability classification using unsupervised on-line visual learning for outdoor robot navigation	Kim et al.	2006	agent	global	1st	stable	no	macro	no	no	exploration	Classification	yes	planning	online	traversability	mathematical		Clustering, Classification	3D pixel information, texture	self-supervised	real robot	original study
Visual object-action recognition: Inferring object affordances from human demonstration	Kjellström et al.	2011	agent	meso	0th	stable	no	micro	no	no	demonstration	Classification	no	not specified	offline	open, pour, hammer	mathematical		Factorial CRF	spatial pyramids of HoG	supervised	benchmark	original study
Physically Grounded Spatio-temporal Object Affordances	Koppula et al	2014	observer	meso	2nd	stable	no	macro	yes	yes	ground truth	Optimization	yes	action selection	offline	general	mathematical		Graphical model, GPR	human pose, feat. w.r.t. skeleton joints / objects	supervised	benchmark	original study
Learning human activities and object affordances from RGB-D videos	Koppula et al.	2013	observer	meso	2nd	stable	yes	micro	yes	yes	ground truth	Classification	yes	action selection	online	general	mathematical	not specified	SVM (MRF, kNN, Particle Filter)	BB, centroid, SIFT	supervised	benchmark	original study
Collision risk assessment for autonomous robots by offline traversability learning	Kostavelis et al.	2012	agent	global	1st	stable	no	micro	no	no	ground truth	Classification	yes	single-/multi-step prediction	offline	traversability	mathematical		SVM	dispartiy maps, hist. of pixel distribution	supervised	real robot	original study
A kernel-based approach to direct action perception	Kroemer et al.	2012	agent	local	1st	stable	no	micro	yes	no	demonstration	Regression	yes	action selection	offline	pouring, grasping	mathematical	not specified	non-parametric surface kernel, kernel logistic regression, DMP	pointclouds	supervised	real robot	original study
A Flexible Hybrid Framework for Modeling Complex Manipulation Tasks	Kroemer et al.	2011	observer	meso	1st	stable	no	macro	yes	yes	hardcoded	Optimization	no	planning	offline	grasping, pushing, striking	mathematical		RL	pose	supervised	real robot	original study
A perceptual system for vision-based evolutionary robotics	Kubota et al.	2003	agent	global	1st	variable	no	micro	no	no	exploration	Optimization	yes	action selection	online	traversability	mathematical		SSGA, clustering	optical flow	unsupervised	real robot	original study
Goal-oriented Dependable Action Selection using Probabilistic Affordance	Lee et al.	2010	agent	meso	1st	stable	no	micro	yes	yes	ground truth	Classification	yes	action selection	offline	general	mathematical		multilayer naive Bayesian classifier	not specified	supervised	real robot	original study
Skill Learning and Inference Framework for Skilligent Robot	Lee et al.	2013	agent	meso	1st	stable	no	micro+macro	yes	yes	demonstration	Optimization	no	action selection	offline	general	mathematical		BN, DMP	trajectories (joints and end-effectors)	supervised	real robot	original study
Foot Placement Selection Using Non-geometric Visual Properties	Lewis et al.	2005	agent	global	1st	stable	no	micro	yes	no	exploration	Classification	yes	planning	online	locomotion	mathematical		NN	color, texture	supervised	simulation+real robot	original study
Affordance-based imitation learning in robots	Lopes et al.	2007	agent	meso	1st	stable	no	micro	yes	no	demonstration+exploration	Optimization	yes	action selection	offline	grasping, tapping, touching	mathematical		BN, RL	shape, color, scale	semi-supervised	real robot	original study
Responding to affordances: Learning and Projecting a Sensorimotor Mapping	MacDorman	2000	agent	meso	1st	stable	no	micro	no	yes	exploration	Classification	no	planning	online	navigation	mathematical		Partition Nets	color	self-supervised	simulation	original study
Multi-model approach based on 3D functional features for tool affordance learning in robotics	Mar et al.	2015	agent	meso	1st	stable+variable	no	micro	yes	no	exploration	Regression	yes	not specified	offline	pulling/dragging	mathematical	not specified	SOM, k-means, GRNN	OMS-EGI (3D)	unsupervised	real robot	original study
Self-supervised learning of grasp dependent tool affordances on the iCub Humanoid robot	Mar et al.	2015	agent	meso	1st	stable+variable	no	micro	no	no	exploration	Classification	yes	single-/multi-step prediction	offline	pulling, dragging	mathematical	not specified	SVM, K-means.	2D geometrical features	self-supervised	simulation+real robot	original study
Extending sensorimotor contingency theory: prediction, planning, and action generation	Maye & Engl	2013	agent	global	1st	stable+variable	no	micro	no	no	exploration	Inference	yes	single-/multi-step prediction	online	traversability	mathematical		SMC, Markov models	not specified	unsupervised	real robot	original study
Better Vision through Manipulation	Metta & Fitzpatrick	2003	agent	meso	1st	stable	no	micro	no	no	exploration	Classification	no	action selection	online	rollability	neural	AIP-F5	Clustering	color	unsupervised	real robot	original study
Affordance Learning Based on Subtask's Optimal Strategy	Min et al	2015	agent	meso	1st	stable+variable	no	micro	no	no	exploration	Inference	no	action selection	online	locomotion	mathematical		HRL	shape	supervised	simulation+real robot	original study
The initial development of object knowledge by a learning robot	Modayil et al.	2008	agent	meso	1st	stable	yes	micro	no	no	exploration	Optimization	yes	planning	online	manipulability	mathematical		clustering, utility functions	shape	unsupervised	real robot	original study
From object-action to property-action: Learning causally dominant properties through cumulative explorative interactions	Mohan et al.	2014	agent	meso	1st	stable	no	micro	no	no	exploration	Classification	yes	planning	online	reach, grasp, push, search	mathematical		SOMs	size, color, shape, world map	unsupervised	real robot	original study
Learning relational affordance models for robots in multi-object manipulation tasks	Moldovan et al.	2012	agent	global	2nd	stable	no	micro	yes	yes	ground truth	Inference	no	single-/multi-step prediction	offline	general	mathematical	not specified	Stastistical Relational Learning	not specified	unsupervised	real robot	original study
Occluded Object Search by Relational Affordances	Moldovan et al.	2014	agent	meso	0th	stable	no	micro	yes	no	hardcoded	Optimization	no	action selection	offline	general	mathematical		BN	geometric properties	supervised	simulation	original study
Learning Grasping Affordances From Local Visual Descriptors	Montesano and Lopes	2009	agent	local	1st	stable	no	micro	no	no	exploration	Inference	yes	action selection	offline	grasping	mathematical	not specified	Bayes	Gaussian, Sobel, Laplacian Filters	unsupervised	real robot	original study
Modelling Affordances Using Bayesian Networks	Montesano et al.	2007	agent	meso	1st	stable	no	micro	no	no	exploration	Inference	no	action selection	offline	general	mathematical	not specified	BN, MCMC	color, shape, size, position; robot gripper pose	unsupervised	real robot	original study
Learning Object Affordances: From Sensory-Motor Coordination to Imitation	Montesano et al.	2008	agent	meso	2nd	stable	no	micro	yes	no	exploration	Inference	yes	single-/multi-step prediction	offline	general	mathematical	not specified	BN, MCMC	convexity, compactness, circleness, squareness	unsupervised	real robot	original study
Affordances, development and imitation	Montesano et al.	2007	agent	meso	1st	stable	no	micro	yes	no	demonstration	Optimization	no	action selection	offline	grasping, taping	mathematical		BN	color, shape, size	supervised	real robot	original study
Case Studies of Applying Gibson’s Ecological Approach to Mobile Robots	Murphy	1999	agent	meso	1st	stable	no	micro	no	no	hardcoded	Classification	no	action selection	offline	docking, path following, picking	mathematical		Hard-coded perceptual affordance detectors	HC perceptual affordance detectors	not specified	real robot	original study
Affordance Estimation For Vision-Based Object Replacement on a Humanoid Robot	Mustafa et al.	2016	agent	meso	2nd	stable	no	micro	yes	no	ground truth	Classification	yes	action selection	offline	general	mathematical	not specified	JointSVM	3D texlets	unsupervised	real robot	original study
Affordance Detection of Tool Parts from Geometric Features	Myers et al.	2015	agent	local	0th	stable	no	micro	no	no	ground truth	Inference	yes	action selection	offline	general	mathematical	not specified	SRF	Depth, SNorm, PCurv, SI+CV	unsupervised	benchmark	original study
Structural Feature Extraction based on Active Sensing Experiences	Nishide et al.	2008	agent	meso	1st	stable	no	micro+macro	no	no	exploration	Regression	no	single-/multi-step prediction	offline	pushing	mathematical		RNNPB, hierarchical NN	shape, motion	supervised	real robot	original study
Active Sensing based Dynamical Object Feature Extraction	Nishide et al.	2008	agent	meso	1st	stable	no	micro+macro	no	no	exploration	Regression	no	single-/multi-step prediction	offline	pushing	mathematical		RNNPB, hierarchical NN	shape, motion	supervised	real robot	original study
Modeling Tool-Body Assimilation using Second-order Recurrent Neural Network	Nishide et al.	2009	agent	global	1st	stable	no	micro+macro	no	yes	exploration	Regression	yes	single-/multi-step prediction	offline	pulling, dragging	mathematical		SOM, Multiple time-scales RNN	SOM object feature from image	supervised	real robot	original study
Tool–Body Assimilation of Humanoid Robot Using a Neurodynamical System	Nishide et al.	2012	agent	meso	1st	stable	no	micro	no	no	exploration	Regression	yes	action selection	offline	manipulability	mathematical		SOM, MTRNN, HNN	SOM output	semi-supervised	real robot	original study
Generation of behavior automaton on neural network	Ogata et al.	1997	agent	global	1st	stable	no	micro	no	no	exploration	Classification	yes	planning	online	traversability	mathematical		SOM, Temporal Sequence Network	SOM output	semi-supervised	simulation	original study
Symbol Generation and Feature Selection for Reinforcement Learning Agents Using Affordances and U-Trees	Oladell et al.	2012	agent	meso	1st	stable	no	micro	yes	no	hardcoded	Optimization	no	action selection	offline	lifting, dropping, stacking	mathematical		MDP	location, shape, color	supervised	simulation	original study
Autonomous acquisition of pushing actions to support object grasping with a humanoid robot	Omrcen et al.	2009	agent	meso	1st	stable	not specified	micro+macro	yes	no	exploration	Optimization	yes	single-/multi-step prediction	offline	grasping, pushing	mathematical		NN	objecet image	supervised	real robot	original study
Reinforcement Learning of Predictive Features in Affordance Perception	Paletta & Fritz	2008	agent	meso	1st	stable+variable	no	micro	no	no	exploration	Classification	yes	planning	online	liftability	mathematical		Q-Learning, k-means	SIFT	supervised	simulation	original study
Perception and Developmental Learning of Affordances in Autonomous Robots	Paletta et al.	2007	agent	local	1st	stable	yes	micro	no	no	ground truth	Optimization	no	action selection	offline	lifting	mathematical		MDP	SIFT, color, shape	supervised	real robot	original study
Affordance-feasible planning with manipulator wrench spaces	Price et al.	2016	agent	meso	1st	stable	no	micro	no	yes	exploration	Classification	yes	planning	offline	grasping	mathematical		BN	wrenches	not specified	simulation+real robot	original study
Bio-inspired Model of Robot Adaptive Learning and Mapping	Ramierz & Widel	2006	agent	global	1st	stable	no	micro	yes	no	exploration	Classification	yes	action selection	online	traversability	neural	hippocampus	RL, Hebbian Learning	color	supervised	real robot	original study
Increasing the Autonomy of Mobile Robots by On-line Learning Simultaneously at Different Levels of Abstraction	Richert et al	2008	agent	meso	1st	stable	no	micro	yes	no	exploration	Classification	yes	action selection	online	traversability	mathematical		RL, Decision Trees	color, distance, angle	supervised	simulation	original study
Action-grounded push affordance bootstrapping of unknown objects	Ridge and Ude	2013	agent	meso	1st	stable	no	micro	no	no	exploration	Classification	yes	single-/multi-step prediction	online	pushing	mathematical	not specified	SOM, LVQ, Hebbian learning, K-means	action-grounded 3D shape	self-supervised	real robot	original study
Self-supervised cross-modal online learning of basic object affordances for developmental robotic systems	Ridge et al.	2010	agent	meso	1st	stable	no	micro	no	no	exploration	Classification	yes	single-/multi-step prediction	online	pushing	mathematical	not specified	SOM, LVQ, Hebbian learning, K-means	2D,3D shape, 2D motion	self-supervised	real robot	original study
Self-supervised Online Learning of Basic Push Affordances	Ridge et al.	2015	agent	meso	1st	stable	no	micro	no	no	exploration	Classification	yes	single-/multi-step prediction	online	pushing	mathematical	not specified	SOM, LVQ, Hebbian learning, K-means	2D,3D shape and motion	self-supervised	real robot	original study
The MACS Project: An Approach to Affordance-Inspired Robot Control	Rome et al.	2008	agent	global	1st	stable	yes	micro	no	no	ground truth	Classification	no	planning	offline	lifting, trabersability	mathematical		Nearest Neighbor	SIFT	supervised	real robot	original study
A Multi-scale CNN for Affordance Segmentation in RGB Images	Roy et al.	2016	agent	global	0th	stable	no	micro	no	no	ground truth	Classification	yes	not specified	offline	walkable, sittable, lyable, and reachable	mathematical		Multi-Scale CNN	RGB+D, surface normals, semantic labels	supervised	benchmark	original study
Learning the Consequences of Actions: Representing Effects as Feature Changes	Rudolph et al.	2010	agent	meso	1st	stable	no	micro	no	no	demonstration	Inference	yes	single-/multi-step prediction	online	general	mathematical		BN	object, world, meta (object-object) features	supervised	simulation	original study
To Afford or Not to Afford: A New Formalization of Affordances Toward Affordance-Based Robot Control	Şahin et al.	2007	agent	meso	1st	stable	no	micro	no	no	exploration	Classification	yes	planning	online	general	mathematical	not specified	SVM, STRIPS	not specified	not specified	real robot	original study
The acquisition of intentionally indexed and object centered affordance gradients: A biomimetic controller and mobile robotics benchmark	Sánchez-Fibla et al.	2011	agent	meso	1st	stable	no	micro	no	no	exploration	Regression	yes	action selection	online	pushing	mathematical		Affordance gradients	shape, position, orientation	unsupervised	real robot	original study
A Logic-based Computational Framework for Inferring Cognitive Affordances	Sarathy & Scheutz	2016	agent	meso	1st	stable+variable	no	micro	no	no	demonstration+exploration	Inference	yes	planning	online	cognitive affordance	mathematical		Logic Programming	visual information	unsupervised	simulation	original study
Bootstrapping the Semantics of Tools: Affordance Analysis of Real World Objects on a Per-part Basis	Schoeler and Wörgötter	2015	agent	local	1st	stable	no	micro	no	no	ground truth	Classification	yes	action selection	offline	general	mathematical	not specified	SVM	SHOT, ESF	unsupervised	benchmark	original study
Bayesian Network Model for Object Concept	Shinchi et al.	2007	agent	meso	0th	stable	no	micro	no	no	demonstration	Inference	yes	not specified	online	general	mathematical		BN	color, contour, barycentric pos., num. of objects	unsupervised	benchmark	original study
Learning and generalization of behavior-grounded tool affordances	Sinapov and Stoytchev	2007	agent	meso	1st	stable+variable	no	micro	no	no	exploration	Classification	yes	single-/multi-step prediction	offline	pulling, dragging	mathematical	not specified	K-NN, decision tree.	changes in raw pixels	self-supervised	simulation	original study
Detecting the functional similarities between tools using a hierarchical representation of outcomes	Sinapov and Stoytchev	2008	agent	meso	1st	stable	no	micro	no	no	exploration	Classification	no	single-/multi-step prediction	online	pulling, dragging	mathematical	not specified	X-Means, Ensemble of C4.5 Decision tree classifiers.	raw pixels, trajectories	self-supervised	simulation	original study
Learning to Detect Visual Grasp Affordances	Song et al.	2016	agent	meso	1st	stable	no	micro	no	no	ground truth	Classification	yes	action selection	offline	grasping	mathematical	not specified	MMR	BB, category, texture	supervised	real robot	original study
Learning Task Constraints for Robot Grasping using Graphical Models	Song et al.	2010	agent	meso	1st	stable	no	micro	yes	no	ground truth	Optimization	yes	action selection	offline	grasping	mathematical		BN (GMM, Multinomial distribution)	size, convexity, grasp pose	supervised	simulation	original study
Visual Grasp Affordances From Appearance-Based Cues	Song et al.	2011	agent	meso	0th	stable	no	micro	yes	no	ground truth	Regression	no	not specified	offline	grasping	mathematical		MMR	local features, HOG	supervised	benchmark	original study
Predicting Human Intention in Visual Observations of Hand/Object Interactions	Song et al.	2013	observer	meso	1st	stable	not specified	micro	yes	no	demonstration	Optimization	no	action selection	offline	grasping	mathematical		BN, GMM, SOM	grasp parameters, dimension	supervised	real robot	original study
Embodiment-Specific Representation of Robot Grasping using Graphical Models and Latent-Space Discretization	Song et al.	2011	observer	meso	1st	stable	not specified	micro	yes	no	demonstration	Optimization	yes	action selection	offline	grasping	mathematical		BN, Gaussian Latent Variable Model	grasp parameters, dimension	supervised	simulation	original study
Task-Based Robot Grasp Planning Using Probabilistic Inference	Song et al.	2015	agent	meso	1st	stable	not specified	micro	yes	no	demonstration	Optimization	yes	action selection	offline	general	mathematical		BN	shape, grasp parameters	supervised	simulation+real robot	original study
Functional object class detection based on learned affordance cues	Stark et al.	2008	agent	meso	0th	stable	no	micro	no	no	demonstration	Classification	yes	action selection	offline	grasping	mathematical	not specified	KDE, Hough transform	k-adjacent segments, ISM	supervised	real robot	original study
Learning the Affordances of Tools Using a Behavior-Grounded Approach	Stoytchev	2008	agent	meso	2nd	stable+variable	no	micro	yes	yes	exploration	Inference	no	action selection	online	grasping	mathematical	not specified	Grounding	position, color	unsupervised	real robot	original study
Behavior-Grounded Representation of Tool Affordances	Stoytchev	2005	agent	meso	1st	stable	no	micro	no	no	exploration	Classification	no	action selection	offline	extend, slide, contract	mathematical		Affordance table	position, color	self-supervised	real robot	original study
Behavior-Grounded Representation of Tool Affordances	Stoytchev	2005	agent	global	1st	stable+variable	no	micro+macro	no	yes	exploration	Regression	no	single-/multi-step prediction	online	pulling, dragging, pushing, grasping	mathematical		Probabilistic lookup table	object postion, tool color	unsupervised	real robot	original study
A Bayesian Approach Towards Affordance Learning in Aritifical Agents	Stramandinoli et al.	2015	agent	meso	1st	stable	no	micro	no	no	exploration	Optimization	no	action selection	offline	general	mathematical	not specified	BN, MLE	not specified	unsupervised	real robot	original study
Learning Visual Object Categories for Robot Affordance Prediction	Sun et al.	2010	agent	meso	1st	stable	no	micro	no	yes	ground truth	Classification	no	planning	offline	locomotion	mathematical		BN, EM, GM, DIRECT	color, edge	supervised	real robot	original study
A model of shared grasp affordances from demonstration	Sweeney and Grupen	2007	agent	meso	1st	stable	no	micro	no	no	demonstration	Inference	yes	action selection	offline	grasping	mathematical	not specified	BN, MLE	moment feature	supervised	real robot	original study
Knowledge Propagation and Relation Learning for Predicting Action Effects	Szedmak et al.	2014	agent	meso	1st	stable	no	micro	no	no	ground truth	Classification	yes	not specified	offline	object	mathematical	not specified	MMMVR	shape, size	supervised	benchmark	original study
Perception driven robotic assembly based on ecological approach	Tagawa et al.	2002	agent	meso	1st	stable	no	micro	no	no	exploration	Inference	yes	action selection	online	general (positive and negative)	mathematical		Genetic Algorithms, State-machines	object postion	unsupervised	simulation	original study
Localizing Handle-like Grasp Affordances In 3D Point Clouds	Ten Pas et al	2016	agent	local	0th	stable	no	micro	no	no	hardcoded	Optimization	yes	action selection	not specified	grasping	mathematical		Importance sampling, quadratic surface fitting	curvature, circle fitting	not specified	real robot	original study
Exploring affordances and tool use on the iCub	Tikhanoff et al.	2013	agent	meso	1st	stable	no	micro	yes	no	exploration	Regression	no	single-/multi-step prediction	online	pulling, dragging	mathematical	not specified	Linear SVM, least squares	SIFT, pull angle, tracked dist.	supervised	real robot	original study
Traversability: A Case Study for Learning and Perceiving Affordances in Robots	Ugur et al.	2010	agent	global	1st	stable	yes	micro	no	no	exploration	Classification	yes	action selection	offline	traversability	mathematical	not specified	SVM	shape, size	self-supervised	real robot	original study
Goal emulation and planning in perceptual space using learned affordances	Ugur et al.	2011	agent	meso	1st	stable	no	micro	no	yes	exploration	Classification	yes	planning	offline	object	mathematical	not specified	X-means, SVM	shape, size	unsupervised	real robot	original study
Staged Development of Robot Skills: Behavior Formation, Affordance Learning and Imitation	Ugur et al.	2015	agent	meso	1st	stable	no	micro	no	yes	demonstration+exploration	Classification	yes	single-/multi-step prediction	offline	object	mathematical	not specified	DTW, X-means, SVM, EM	shape, size	unsupervised	real robot	original study
Emergent structuring of interdependent affordance learning tasks using intrinsic motivation and empirical feature selection	Ugur et al.	2016	agent	meso	1st	stable	yes	micro	no	no	ground truth	Classification	yes	action selection	online	object	mathematical	not specified	SVM, intrinsic motivation	shape, size	supervised	benchmark	original study
Bottom-Up Learning of Object Categories, Action Effects and Logical Rules: From Continuous Manipulative Exploration to Symbolic Planning	Ugur et al.	2015	agent	meso	1st	stable+variable	no	micro	no	yes	exploration	Classification	yes	planning	offline	object	mathematical	not specified	SVM, C4.5 Decision tree, X-means, PDDL	shape, size	unsupervised	real robot	original study
AfNet: The Affordance Network	Varadarajan et al.	2012	agent	local	0th	stable	no	micro	no	no	ground truth	Classification	yes	action selection	offline	general	mathematical		not specified	superquadrics	supervised	benchmark	original study
AfRob: The Affordance Network Ontology for Robots	Varadarajan et al.	2012	agent	local	0th	stable	no	micro	no	no	ground truth	Classification	yes	action selection	offline	general	mathematical		not specified	gradient image, superquadrics	supervised	benchmark	original study
Predicting slippage and learning manipulation affordances through Gaussian Process regression	Vina et al	2013	agent	meso	1st	stable	no	micro	yes	no	exploration	Regression	yes	planning	offline	grasping	mathematical		GP	hand-object relative pose	supervised	real robot	original study
Robot Learning and Use of Affordances in Goal-directed Tasks	Wang et al.	2013	agent	global	1st	variable	no	micro	no	no	exploration	Optimization	no	action selection	online	moveability	mathematical		Extended classifier system (XCS)	color, size	semi-supervised	real robot	original study
An Entropy-Based Approach to the Hierarchical Acquisition of Perception-Action Capabilities	Windridge et al.	2008	agent	meso	1st	stable	no	micro	yes	yes	exploration	Optimization	yes	action selection	online	sorting	mathematical		SGD	image point entropy	unsupervised	simulation	original study
A novel formalization for robot cognition based on Affordance model	Yi et al.	2012	agent	meso	1st	stable	no	micro	no	no	exploration	Inference	yes	action selection	online	carryable, stackable, liftable, moveable	mathematical		First-order logic, analysis functions	color, size	unsupervised	simulation	original study
Fill and Transfer: A Simple Physics-based Approach for Containability Reasoning	Yu et al.	2015	agent	meso	0th	stable	no	micro	yes	no	ground truth	Optimization	yes	action selection	offline	containability	mathematical		smoothing-based optimization, Gaussian sampling	voxels	supervised	benchmark	original study
The learning of adjectives and nouns from affordance and appearance features	Yuruten et al	2013	agent	meso	1st	stable	yes	micro	no	no	exploration	Classification	yes	language	offline	manipulation	mathematical	-	ReliefF, SVM	3D shape, size	self-supervised	benchmark+real robot	original study
Learning Adjectives and Nouns from Affordances on the iCub Humanoid Robot	Yuruten et al	2012	agent	meso	1st	stable	yes	micro	no	no	exploration	Classification	no	language	offline	manipulation	mathematical	-	ReliefF, SVM, Growing Neural Gas	3D shape, size	self-supervised	real robot	original study
Reasoning about Object Affordances in a Knowledge Base Representation	Zhu et al.	2014	agent	meso	1st	stable	no	micro	no	no	hardcoded	Inference	yes	single-/multi-step prediction	offline	general	mathematical		Markov Logic Network	pose, human-object pose info	supervised	benchmark	original study
Understanding tools: Task-oriented object modeling, learning and recognition	Zhu et al.	2015	agent	local	1st	stable	no	micro	yes	no	demonstration	Optimization	yes	action selection	online	tool-use	mathematical		SVM, ranking function	material, volume, mass	supervised	benchmark	original study
Learning the semantics of object–action relations by observation	Aksoy et al.	2011	agent	meso	0th	stable	no	micro	no	no	demonstration	Classification	no	not specified	offline	Moving Object, Making Sandwich, Filling Liquid, and Opening Book	mathematical		Semantic Object-Hand and Object-Object relations	Color and Depth	supervised	benchmark	Eren Aksoy (09/2017)
Model-free incremental learning of the semantics of manipulation actions	Aksoy et al.	2015	agent	meso	0th	stable	no	micro	no	no	demonstration	Classification	no	not specified	online	Pushing, hiding, cutting, chopping, uncovering, putting	mathematical		Semantic Object-Hand and Object-Object relations	Color and Depth	unsupervised	benchmark	Eren Aksoy (09/2017)
Object-Action Complexes: Grounded Abstractions of Sensorimotor Processes	Krüger et al.	2011	agent	meso	1st	stable	no	micro+macro	no	yes	exploration	Regression	yes	planning	online	pushing, grasping	mathematical		NN, KDE, Sampling	co-planar contours, location, ECV	supervised	real robot	Tamim Asfour (09/2017)
What can I do with this tool? Self-supervised learning of tool affordances from their 3D geometry	Mar et al.	2017	agent	meso	1st	stable	no	micro	no	no	exploration	Regression	yes	action selection	offline	tool-use	mathematical		SOM, regression	OMS-EGI	self-supervised	simulation+real robot	Tanis Mar (09/2017)
Towards a Hierarchy of Loco-Manipulation Affordances	Kaiser et al.	2016	agent	global	1st	stable	no	macro	no	yes	ground truth	Optimization	yes	action selection	not specified	loco-manipulation	mathematical		sampling, decision functions	shape, distance, orientation	not specified	simulation+real robot	Peter Kaiser (09/2017)
A modular Dynamic Sensorimotor Model for affordances learning, sequences planning and tool-use	Braud et al.	2017	agent	local	1st	stable	no	micro	no	no	demonstration+exploration	Optimization	yes	single-/multi-step prediction	online	tool-use	mathematical		Sensorimotor Law Encoders/Simualtor, Dynamic Sensorimotor Model	sensor and motor readings	self-supervised	simulation+real robot	Alexandre Pitti (10/2017)
Detecting object affordances using Convolutional Neural Networks	Nguyen et al.	2016	agent	meso	0th	stable	no	micro	no	no	ground truth	Classification	yes	action selection	offline	general	mathematical		CNN	RGB, depth	supervised	benchmark+real robot	Philipp Zech (10/2017)
Object-Based Affordances Detection with Convolutional Neural Networks and Dense Conditional Random Fields	Nguyen et al.	2017	agent	meso	0th	stable	no	micro	no	no	ground truth	Classification	yes	action selection	offline	general	mathematical		CNN, CRF	RGB , depth	supervised	benchmark+real robot	Philipp Zech (10/2017)
Iterative affordance learning with adaptive action generation	Maestre et al.	2017	agent	meso	1st	stable	no	micro	no	no	exploration	Inference	yes	action selection	online	pushing	mathematical		BN	position	self-supervised	simulation+real robot	Philipp Zech (10/2017)
Learning to Segment Affordances	Lübbecke and Wörgötter	2017	agent	global	0th	stable	no	micro	no	no	ground truth	Classification	yes	not specified	offline	general	mathematical		CNN	RGB, object segments, object parts	supervised	benchmark	Philipp Zech (10/2017)
Discovering and Manipulating Affordances	Chavez-Garcia et al.	2016	agent	meso	1st	stable	no	micro	no	no	exploration	Optimization	yes	not specified	offline	pushing	mathematical		clustering, BN	supervoxels, forces	supervised	real robot	Mihai Andries (10/2017)
Title	Authors	Year	Perception Perspective	Level	Order	Temporality	Selective Attention	Abstraction	Competitive	Chaining	Acquisition	Prediction	Generalization	Exploitation	Learning	Kind	Abstraction	Brain Areas	Method	Features	Training	Evaluation	Data added

Please cite as follows:
@Article{Zech-2017-AB,
  title = {{Computational models of affordance in robotics: a taxonomy and systematic classification}},
  author = {Zech, Philipp and Haller, Simon and Rezapour Lakani, Safoura and Ridge, Barry and Ugur, Emre and Piater, Justus},
  journal = {{Adaptive Behavior}},
  year = 2017,
  month = 10,
  volume = 25,
  number = 5,
  pages = {235--271},
  publisher = {SAGE},
  doi = {10.1177/1059712317726357},
  url = {https://iis.uibk.ac.at/public/papers/Zech-2017-AB.pdf}
}

Updates:

Added papers requested by Mihai Andries (10/2017)
Added errata for original Adaptive Behavior Publication (10/2017)
Added papers requested by Philipp Zech (10/2017)
Added papers requested by Alexandre Pitti (10/2017)
Added papers requested by Peter Kaiser (09/2017)
Added papers requested by Tanis Mar (09/2017)
Added papers requested by Tamim Asfour (09/2017)
Added papers requested by Eren Aksoy (09/2017)
Initial release of this page (08/2017)