Skip to content. | Skip to navigation

Personal tools
Intrinsically Motivated
Cumulative Learning
Versatile Robots
Document Actions

Abstracts of keynote speeches, and Posters

Back to the web-page of the IMOD-2013 Workshop

Abstracts of invited talks

Cyriel Pennartz

(Department Cognitive and Systems Neuroscience, Swammerdam Institute for Life Sciences, Center for Neuroscience CSCA, Cognitive Science Center Amsterdam, Faculty of Science, University of Amsterdam)
Hippocampus, striatum and beyond: interweaving motivation with action, memory and perception
In this talk I will begin with addressing different kinds of motivation and its sources, from homeostatic drives to exploration of novel environments and the pursuit of cues and contexts that are predictive of beneficial outcome. The topic of cue- and context-based reward predictions will be deepened by looking at the systems neurophysiology of the hippocampus and ventral striatum, which have been implicated in episodic memory and learned motivated behavior, but which are nonetheless intimately connected. Next to the role of this system in place-reward associative learning we will consider new data on "remapping" phenomena in these two structures, suggesting a novel mechanism for incorporating motivational signalling in high-dimensional, model-based coding.
A further source of afferent input to the striatum stems from the prefrontal cortex, and we will especially scrutinize the orbitofrontal cortex, which has been linked to flexible representations of stimulus-outcome associations. In the course of associative learning, groups of neurons discriminate progressively better between stimuli that are coupled to either a positive (rewarding) outcome or a negative (punishing) outcome. Because the mechanism underlying this development of spike-pattern discriminability is unknown, we investigated the causal role of NMDA receptors in this form of plasticity, using tetrodes combined with local perfusion of an NMDA receptor blocker in rat orbitofrontal cortex. The results confirm that NMDA receptors are instrumental in this reward-dependent form of neural plasticity, and shed light on their role in neural synchronization.
A final topic will be to ask how pervasive the effects of motivation and reward-dependent learning are throughout the brain, taking primary visual cortex as a case study. We will discuss 2-photon imaging evidence showing that reward information not only impinges on primary sensory areas, but also has long-lasting effects on visual tuning curves of specific cell assemblies.

Daniel Polani

(Department of Computer Science, University of Hertfordshire, UK)
The Cognitive Imprint of Environment and Intrinsic Motivation
A challenge in understanding (in biology) and modeling (in AI) generic cognitive architectures is to understand where sensible better-than-random behaviours could possible arise from. In recent years, an intuition is arising that, different from theoretical scenarios, in the real world cognitive mechanisms do not have to contend with worst-case scenarios. The real world is highly structured and so is the link between, say, the cognitive architecture of an organism and its environment, its embodiment. This means that, in a certain way, the cognitive dynamics of an organism can be expected to reflect the environmental/ embodimental structure it is situated in and this, in turn, provides a route to produce environment-adapted mechanisms for intrinsic motivation. Information theory has proven to be an effective language in which to give these notions a precise meaning, and the present talk will present a number of notions, techniques and observations based on informational optimization principles in which it becomes clearer how the environment can imprint itself on the cognitive architecture to provide routes towards intrinsic motivation.

 

Eugenia Polizzi di Sorrentino / Fabrizio Taffoni

(Unit of Cognitive Primatology and Primate Centre, Institute of Cognitive Sciences and Technologies, CNR, Rome & Universita' Campus Bio-Medico, Rome)
A mechatronic Platform for empirical experiments on intrinsic motivations and skill acquisition. Experiments with children and monkeys
The autonomous acquisition of new skills and knowledge is one of the most astonishing capacities observed in humans and other animal species. However, the driving force that shapes this process is still unknown. Children seem to acquire new skills and know-how in a continuous and open-ended manner. From an evolutionary point of view, the same processes seem to occur in our close relatives, apes and monkeys. How (non) human primates learn to use these skills in a different context to reach a specific goal is unknown. It has been suggested that intrinsic motivation, described as a drive that leads exploratory actions "for their own sake", may play an important role in the acquisition of skills. In order to understand whether spontaneous exploration drives versatile learning, a novel tool for behavioural analysis (“mechatronic board”) was build. The board, specifically designed to allow inter-species comparative research, allowed testing whether exploratory actions (not instrumental to achieve any specific goal) improve subjects ability in solving a subsequent goal-directed task requiring the proficiency acquired during previous exploration. Here we provide the results of a set of experiments run with children and capuchin monkeys (Cebus apella), a primate species well known for being "curious" and highly manipulative. 

 

Emre Duzel

(Otto-von-Guericke-Universitat Magdeburg, Germany)

Gianluca Baldassarre

(Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, CNR, Rome)

Biological and robotic perspectives on intrinsic motivations

Intrinsic motivations are gaining an increasing attention within cognitive sciences for their importance for mammals and for their potential to drive open-ended learning in machines and robots. However, a clear identification of the core functions and mechanisms of intrinsic motivations is still lacking. This presentation aims to contribute to clarify what intrinsic motivations are from both a biological and a computational perspective. To this purpose, it first shows how intrinsic motivations can be defined contrasting them to extrinsic motivations in terms of function: extrinsic motivations guide learning and drive behaviour on the basis of how this directly affect fitness increase; instead, intrinsic motivations guide learning and drive behaviour on the basis of how this leads to acquire knowledge and skills that contribute to increase fitness only in a later stage. Given this functional difference, extrinsic motivations generate learning/motivational signals on the basis of events involving body homeostatic regulations (in robots: ''involving the task''), whereas intrinsic motivations generate learning/motivational signals based on events taking place within the brain itself (in robots: ''within the controller''). These different functions are supported by several different brain/computational mechanisms: those related to intrinsic motivations are first classified and illustrated theoretically, and then are exemplified on the basis of biological evidence and some robotic examples.

 

Giorgio Metta

(iCub Facility, Italian Institute of Technology, Genova, Italy)
Learning new skills on the iCub
In this talk, I will cover some recent work that uses the iCub to learn object recognition on the fly, exploit binocular vision to plan approximate grasping of the recognized object and combine it with affordance estimation in tool use tasks. The typical scenario includes a benevolent teacher that interact with the robot by showing objects and naming them. We employ state of the art machine learning and analyze online performance of various techniques.

Jochen Triesh

(Frankfurt Institute for Advanced Studies, J.W. Goethe University, Frankfurt am Main, Germany)
Intrinsically Motivated Learning in Active Perception
The goal of perceptual systems is to provide useful knowledge about the environment and to encode this information efficiently. As such, perception is an active process that often involves the movement of sense organs such as the eyes. This active nature of perception has typically been neglected in popular theories describing how nervous systems learn sensory representations. Here we present an approach for intrinsically motivated learning during active perception that treats the learning of sensory representations and the learning of movements of the sense organs in an integrated manner. In this approach, a generative model learns to encode the sensory data while a reinforcement learner directs the sense organs so as to make the generative model work as effectively as possible. To this end, the reinforcement learner receives an intrinsic reward signal that measures the encoding quality currently obtained by the generative model. In the context of binocular vision, the approach is shown to lead to a self-calibrating stereo vision system that learns a representation for binocular disparity while at the same time learning proper vergence eye movements to fixate objects. The approach is quite general and can be applied to other types of eye movements any may be extended to different sensory modalities. Somewhat surprisingly, the approach also offers a new perspective on the development of imitation abilities.

Juergen Schmidhuber

(Scuola Universitaria Professionale della Svizzera Italiana, Istituto Dalle Molle sull'Intelligenza Artificiale, Lugano, Switzerland):
PowerPlay
Most of computer science focuses on automatically solving given computational problems. I focus on automatically inventing or discovering problems in a way inspired by the playful behavior of animals and humans, to train a more and more general problem solver from scratch in an unsupervised fashion. Consider the infinite set of all computable descriptions of tasks with possibly computable solutions. The novel algorithmic framework PowerPlay (2011) continually searches the space of possible pairs of new tasks and modifications of the current problem solver, until it finds a more powerful problem solver that provably solves all previously learned tasks plus the new one, while the unmodified predecessor does not. Wow-effects are achieved by continually making previously learned skills more efficient such that they require less time and space. New skills may (partially) re-use previously learned skills. PowerPlay's search orders candidate pairs of tasks and solver modifications by their conditional computational (time & space) complexity, given the stored experience so far. The new task and its corresponding task-solving skill are those first found and validated. The computational costs of validating new tasks need not grow with task repertoire size. PowerPlay's ongoing search for novelty keeps breaking the generalization abilities of its present solver. This is related to Goedel's sequence of increasingly powerful formal theories based on adding formerly unprovable statements to the axioms without affecting previously provable theorems. The continually increasing repertoire of problem solving procedures can be exploited by a parallel search for solutions to additional externally posed tasks. PowerPlay may be viewed as a greedy but practical implementation of basic principles of creativity.
With Rupesh Kumar Srivastava and Bas Steunebrink, I describe first experiments with PowerPlay. A self-delimiting recurrent neural network SLIM RNN (2012) is used as a general computational problem solving architecture. Its connection weights can encode arbitrary, self-delimiting, halting or non-halting programs affecting both environment (through effectors) and internal states encoding abstractions of event sequences.  Our PowerPlay-driven SLIM RNN learns to become an increasingly general solver of self-invented problems, continually adding new problem solving procedures to its growing skill repertoire. We identify interesting, emerging, developmental stages of our open-ended system. We also show how it automatically self-modularizes, frequently re-using code for previously invented skills, always trying to invent novel tasks that can be quickly validated because they do not require too many weight changes affecting too many previous tasks.

Kevin O'Reagan

(Laboratoire Psychologie de la Perception - Institut Paris Descartes de Neurosciences et Cognition, Paris, France)
Why is it so hard to use a rake?
Though in many ways very adept at grasping and manipulating objects, a 1 year old child is unable to understand that an elongated tool like a rake can be used to obtain an out-of-reach object. It takes a whole year of experience with tools, until age 2 years, for the child to understand that tools can extend its reach. The child also does not seem to be able to benefit from observing other people performing the task. On the basis of a variety of experiments we have done with 9-24 month-old children, I will speculate on possible ways that the children’s difficulties and their surprisingly gradual learning might be understood and modelled.

Mark Lee and James Law

(Department of Computer Science, University of Aberystwyth, UK)
A psychology grounded approach for longitudinal development in cognitive robotics
A major challenge in robotics is the ability to learn, from novel experiences, new behaviour that is useful for achieving new goals and skills.  For autonomous systems such learning must be entirely unsupervised, thus ruling out preprograming or extensive training. It must also be cumulative and incremental, as complex skills are built on top of primitive skills, both of which are acquired through phases of embodied interaction with the environment. Additionally, such learning must be driven by intrinsic motivation because formative experience is gained through autonomous activity, even in the absence of extrinsic goals or tasks.  We present an approach to these issues through modelling and robotic implementations inspired by the manifest learning displayed by the human new-born. We describe an approach to developmental learning based on staged growth of competencies and sensorimotor grounding, and we illustrate this methodology through results from a range of experiments.

Martin McGuinnity

(School of Computing and Intelligent Systems, University of Ulster, Londonderry, UK)
Novelty-based Intrinsic Motivations for Driving Robot Learning
This talk will discuss how intrinsic motivations in humans may be emulated in robotic systems and used as a vehicle for cumulative learning. Having briefly summarized the biological understanding of intrinsic motivation, the role of novelty detection will be explained and utilized as a “curiosity” driven technique for learning. Initial work on visual based perceptual learning and its progression to affordances learning and skills development will be discussed. Finally an example integration of a biologically plausible model for intrinsic motivation with action learning on a physical PR2 robot will be presented.

Minoru Asada

(Department of Adaptive Machine Systems, Graduate School of Engineering, Osaka University, Japan)

Paul Verschure

(Catalan Institute of Advanced Research (ICREA), Technology Department, Universitat Pompeu Fabra, Barcelona, Spain)

Peter Redgrave & Kevin Gurney

(Department of Psychology, University of Sheffield, UK)
The acquisition of novel behaviour: Inspiration from neuroscience
When biological systems (especially mammals) come into this world, comparatively few of their actions are hard wired.  To distinguish purposive actions from early ‘motor babbling’ I will define an action as a single, or series of behavioural responses that are instrumental in causing a predicted outcome.  However, the potentially independent causal components of behaviour (what, where, when and how), the order in which components are assembled, together with their consequent outcomes have to be learnt through trial and error.  This form of learning has been associated with one of the fundamental computing systems of the vertebrate brain, the basal ganglia. I will outline the basic functional neurobiology of the basal ganglia and describe how the operation of a particular component – sensory-evoked dopamine reinforcement – could be used to identify causal associations between behaviour and outcome.  I will argue that the basal ganglia architecture may be used to serve both reward and intrinsically motivated (curiosity-based) learning of novel actions.

 

Kevin Gurney

(Department of Psychology, University of Sheffield, UK)
Computational models of action discovery in animals
How can animals acquire a repertoire of actions enabling the achievement of their goals? Moreover, how can this be done spontaneously without the animal being instructed, or without having some overt, primary reward assigned to successful learning?  The relation between actions and outcomes are presumed to be held in internal models, encoded in associative neural networks. In order for these associations to be learned, representations of the motor action, sensory context, and the sensory outcome must be repeatedly activated in the relevant neural systems. This requires a transient change in the action selection policy of the agent, so that the to-be-learned action is selected more often than other competing actions; we dub this policy change - 'repetition bias' . A key component in this scheme is a set of sub-cortical nuclei - the basal ganglia. There is evidence to suggest the basal ganglia may be subject to reinforcement learning, with phasic activity in midbrain dopamine neurons constituting a reinforcement signal. We propose that this signal encodes a sensory prediction error, thereby suggesting how learning may be intrinsically motivated by exploration of the environment. These ideas have recently been quantified in a model of intrinsically motivated action learning in basal ganglia, and tested in a simple autonomous agent whose behaviour is constrained to mimic that of rats in an in vivo experiment. The model can account for much of the in vivo data, and shows a complex interplay of mechanisms that we believe are responsible for repetition bias and biological action discovery.

Posters

 

Rodolfo Marraffa, Valerio Sperati, Daniele Caligiore, Jochen Triesch and Gianluca  Baldassarre

(Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, CNR, Rome, Italy)
A bio-inspired attention model of anticipation in gaze contingency experiments with infants

Vieri Giuliano Santucci, Gianluca Baldassarre and Marco Mirolli

(Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, CNR, Rome, Italy)
Intrinsic motivation mechanisms for competence acquisition

Valentina Ceravolo, Emilio Cartoni, Giovanni Pezzulo, Gianluca Baldassarre

(Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, CNR, Rome, Italy)
Bayesian models of goal-directed behaviour: extrinsic and novelty-based motivations

Emilio Cartoni, Francesco  Mannella, Stefano Puglisi Allegra Gianluca Baldassarre

(Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, CNR, Rome, Italy; Univerista' "La Sapienza", Rome, Italy)
A bayesian model of goal-directed behavior for a pavlovian-instrumental transfer hypothesis

Vincenzo Fiore, Valerio Sperati, Francesco Mannella, Kevin Gurney, Ray Dolan, Gianluca Baldassarre

(Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, CNR, Rome, Italy)
Keep focussing: striatal dopamine multiple functions resolved in a single mechanism tested on the iCub robot

Armando Romani, Francesco Mannella, Gianluca Baldassarre

(Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, CNR, Rome, Italy)
A computational model of novelty detection in hippocampus based on the comparator hypothesis

Emmet Kerr and Martin McGinnity

(University of Ulster, UK)
Object Exploration and Identification System using Simple Force Sensors

James Connolly and Joan Condell

(University of Ulster, UK)
Robotic ShadowHand in Range of Motion Calculation for Rheumatoid Arthritis Rehab Glove Development

Inaki Rano and Chris Burbridge

(University of Ulster, UK)
Two dimensional Motion Camouflage for Unicycle Robots

Cormac Duffy and Martin McGinnity

(University of Ulster, UK)
Learning Properties of Objects through Simulated Interactions in the iCUB Simulator

Kirill Makukhin, Scott Bolland and Marcus Gallagher

(The University of Queensland)
The emergence of competences in a system with a biologically plausible mechanism of intrinsic motivation

Kevin Earland, Patricia Shaw, James Law, and Mark Lee

(University of Aberystwyth, UK)
Overlapping structures in sensorimotor mappings

Varun Raj Kompella, Marijn Stollenga and Juergen Schmidhuber

(Scuola Universitaria Professionale della Svizzera Italiana, Istituto Dalle Molle sull'Intelligenza Artificiale, Lugano, Switzerland)
Curious Dr. MISFA says IM-CLeVeR!

Hung Ngo, Matthew Luciw, Alexander Forster, Juergen Schmidhuber

(Scuola Universitaria Professionale della Svizzera Italiana, Istituto Dalle Molle sull'Intelligenza Artificiale, Lugano, Switzerland)
Confidence-Based Progress-Driven Self-Generated Goals for Skill Acquisition in Developmental Robots

Pramod Chandrashekhariah and Jochen Triesch

(Stiftung Frankfurt Institut for Advanced Studies)
A curious robot vision system

Luca Lonini, Zhao Yu, Pramod Chandrashekhariah, Bert Shi and Jochen Triesch

(Stiftung Frankfurt Institut for Advanced Studies)
Autonomous Learning of Active Multi-scale Binocular Vision

Jan Hendrik Metzen and Frank Kirchner

(Robotics Research Group, University of Bremen, Bremen, Germany and Robotics Innovation Center, German Research Center for Artificial Intelligence (DFKI), Bremen, Germany)
Incremental Learning of Skill Collections based on Intrinsic Motivation

Dario Di Nocera, Alberto Finzi, Silvia Rossi and Mariacarla Staffa

(Universita' di Napoli "Federico II", Italy)
The Role of Intrinsic Motivations in Attention Shifting

Fabrizio Taffoni, Valentina Focaroli, E. Tamila, Domenico Formica, Eugenio Guglielmelli, Flavio Keller
Eugenia Polizzi di Sorrentino, Gloria Sabbatini, Valentina Truppa, Elisabetta Visalberghi

(Universita' Campus Bio-Medico, Rome, Italy)
(Unit of Cognitive Primatology and Primate Centre, Institute of Cognitive Sciences and Technologies, CNR, Rome)
Learning strategies mediated by intrinsic motivation: an experimental study on children and monkeys

Martin Thirkettle, Peter Redgrave, Kevin Gurney, Tom Walton, Ashvin Shah, Tom Stafford

(University of Sheffield, UK)
Cortical masking and action learning

Jen Lewis, Jon Chambers, Pete Redgrave, Kevin Gurney

(University of Sheffield, UK)
Flexible sequential action selection in a computational model of multiple basal ganglia-thalamocortical loops

Alex Cope, Jon Chambers, Kevin Gurney

(University of Sheffield, UK)
A Systems Integration Approach to Creating Embodied Biomimetic Models of Active Vision

Cornelius Glackin, Christoph Salge, Daniel Polani

(University of Hertfordshire, UK)
Intrinsically Motivated Leg Motion

Anna Lisa Ciancio, Loredana Zollo, Gianluca Baldassarre, Daniele Caligiore, Eugenio Guglielmelli

(Universita' Campus Bio-Medico, Rome, Italy; Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, CNR, Rome, Italy)
A bio-inspired hierarchical neural architecture for dexterous robotic manipulation: a comparative study

Daniele Caligiore, Magda Mustile, Daniele Cipriani, Maria de Marsico, Jochen Triesh, Gianluca Baldassarre

(Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, CNR, Rome, Italy; Frankfurt Institute for Advanced Studies, Frankfurt, Germany)
Intrinsic Motivations for learning visual exploration: a gaze-contingency experiment.

Masaki Ogino, Akihiko Nishikawa and Minoru Asada  

(Faculty of Informatics, Kansai University; Graduate School of Engineering, Osaka University)
Motivation model for interaction between parent and child based on needs for relatedness

Clement Moulin-Frier, Sao Mai Nguyen

(INRIA, Institut National de Recherche en Informatique et en automatique, Bordeaux, France)
The role of intrinsic motivations in early vocal development: a computational study