By Type

Articles

  • [PDF] P. Sequeira, F. S. Melo, and A. Paiva, “Emergence of emotional appraisal signals in reinforcement learning agents,” Autonomous Agents and Multi-Agent Systems, vol. 29, iss. 4, p. 537–568, 2015.

    The positive impact of emotions in decision-making has long been established in both natural and artificial agents. In the perspective of appraisal theories, emotions complement perceptual information, coloring our sensations and guiding our decision-making. However, when designing autonomous agents, is emotional appraisal the best complement to the perceptions? Mechanisms investigated in affective neuroscience provide support for this hypothesis in biological agents. In this paper, we look for similar support in artificial systems. We adopt the intrinsically motivated reinforcement learning framework to investigate different sources of information that can guide decision-making in learning agents, and an evolutionary approach based on genetic programming to identify a small set of such sources that have the largest impact on the performance of the agent in different tasks, as measured by an external evaluation signal. We then show that these sources of information: (i) are applicable in a wider range of environments than those where the agents evolved; (ii) exhibit interesting correspondences to emotional appraisal-like signals previously proposed in the literature, pointing towards our departing hypothesis that the appraisal process might indeed provide essential information to complement perceptual capabilities and thus guide decision-making.

    @article{sequeira2014jaamas,
    Abstract = {The positive impact of emotions in decision-making has long been established in both natural and artificial agents. In the perspective of appraisal theories, emotions complement perceptual information, coloring our sensations and guiding our decision-making. However, when designing autonomous agents, is emotional appraisal the best complement to the perceptions? Mechanisms investigated in affective neuroscience provide support for this hypothesis in biological agents. In this paper, we look for similar support in artificial systems. We adopt the intrinsically motivated reinforcement learning framework to investigate different sources of information that can guide decision-making in learning agents, and an evolutionary approach based on genetic programming to identify a small set of such sources that have the largest impact on the performance of the agent in different tasks, as measured by an external evaluation signal. We then show that these sources of information: (i) are applicable in a wider range of environments than those where the agents evolved; (ii) exhibit interesting correspondences to emotional appraisal-like signals previously proposed in the literature, pointing towards our departing hypothesis that the appraisal process might indeed provide essential information to complement perceptual capabilities and thus guide decision-making.},
    Author = {Sequeira, Pedro and Melo, Francisco S. and Paiva, Ana},
    Date-Modified = {2017-05-09 22:26:51 +0000},
    Doi = {10.1007/s10458-014-9262-4},
    Issn = {1387-2532},
    Journal = {Autonomous Agents and Multi-Agent Systems},
    Keywords = {Emotions, Appraisal Theory, Intrinsic motivation, Genetic programming, Reinforcement learning},
    Number = {4},
    Pages = {537--568},
    Title = {{Emergence of emotional appraisal signals in reinforcement learning agents}},
    Volume = {29},
    Year = {2015},
    Bdsk-Url-1 = {http://dx.doi.org/10.1007/s10458-014-9262-4}}

  • [PDF] P. Sequeira, F. S. Melo, and A. Paiva, “Learning by appraising: an emotion-based approach to intrinsic reward design,” Adaptive Behavior, vol. 22, iss. 5, p. 330–349, 2014.

    In this paper, we investigate the use of emotional information in the learning process of autonomous agents. Inspired by four dimensions that are commonly postulated by appraisal theories of emotions, we construct a set of reward features to guide the learning process and behaviour of a reinforcement learning (RL) agent that inhabits an environment of which it has only limited perception. Much like what occurs in biological agents, each reward feature evaluates a particular aspect of the (history of) interaction of the agent history with the environment, thereby, in a sense, replicating some aspects of appraisal processes observed in humans and other animals. Our experiments in several foraging scenarios demonstrate that by optimising the relative contributions of each reward feature, the resulting “emotional” RL agents perform better than standard goal-oriented agents, particularly in consideration of their inherent perceptual limitations. Our results support the claim that biological evolutionary adaptive mechanisms such as emotions can provide crucial clues in creating robust, general-purpose reward mechanisms for autonomous artificial agents, thereby allowing them to overcome some of the challenges imposed by their inherent limitations.

    @article{sequeira2014adb,
    Abstract = {In this paper, we investigate the use of emotional information in the learning process of autonomous agents. Inspired by four dimensions that are commonly postulated by appraisal theories of emotions, we construct a set of reward features to guide the learning process and behaviour of a reinforcement learning (RL) agent that inhabits an environment of which it has only limited perception. Much like what occurs in biological agents, each reward feature evaluates a particular aspect of the (history of) interaction of the agent history with the environment, thereby, in a sense, replicating some aspects of appraisal processes observed in humans and other animals. Our experiments in several foraging scenarios demonstrate that by optimising the relative contributions of each reward feature, the resulting ``emotional'' RL agents perform better than standard goal-oriented agents, particularly in consideration of their inherent perceptual limitations. Our results support the claim that biological evolutionary adaptive mechanisms such as emotions can provide crucial clues in creating robust, general-purpose reward mechanisms for autonomous artificial agents, thereby allowing them to overcome some of the challenges imposed by their inherent limitations.},
    Author = {Sequeira, P. and Melo, F. S. and Paiva, A.},
    Date-Modified = {2017-05-09 21:53:32 +0000},
    Doi = {10.1177/1059712314543837},
    Issn = {1741-2633},
    Journal = {Adaptive Behavior},
    Month = {Oct},
    Number = {5},
    Pages = {330--349},
    Title = {{Learning by appraising: an emotion-based approach to intrinsic reward design}},
    Volume = {22},
    Year = {2014},
    Bdsk-Url-1 = {http://dx.doi.org/10.1177/1059712314543837}}

In Proceedings

  • [PDF] P. Sequeira, S. Mascarenhas, F. S. Melo, and A. Paiva, “The Development of Cooperation in Evolving Populations through Social Importance,” in Proceedings of the 5th Joint International Conference on Development and Learning and on Epigenetic Robotics, Providence, Rhode Island, 2015, p. 308–313.

    Several agent-based frameworks have been proposed to investigate the possible reasons that lead humans to act in the interest of others while giving up individual gains. In this paper we propose a novel framework for analyzing this phenomenon based on the notions of social importance (SI) and local discrimination. We analyze such mechanism in the context of a “favors game” where a recipient agent can “claim” a favor to a donor agent, which may in turn “confer” its request at the expense of a certain cost. We perform several agent-based simulations and study both the conditions under which cooperation occurs and the dynamics of the relationships formed within a population. The results of our study indicate that the SI mechanism can promote cooperation in populations where all individuals share a common social predisposition towards the favors game, and also in initially mixed-strategy populations evolving by means of mutation and natural selection. We also show that the framework predicts the emergence of a conservative strategy that makes individuals to be “cautious” when interacting with “acquaintances”.

    @inproceedings{sequeira2015icdl,
    Abstract = {Several agent-based frameworks have been proposed to investigate the possible reasons that lead humans to act in the interest of others while giving up individual gains. In this paper we propose a novel framework for analyzing this phenomenon based on the notions of social importance (SI) and local discrimination. We analyze such mechanism in the context of a ``favors game'' where a recipient agent can ``claim'' a favor to a donor agent, which may in turn ``confer'' its request at the expense of a certain cost. We perform several agent-based simulations and study both the conditions under which cooperation occurs and the dynamics of the relationships formed within a population. The results of our study indicate that the SI mechanism can promote cooperation in populations where all individuals share a common social predisposition towards the favors game, and also in initially mixed-strategy populations evolving by means of mutation and natural selection. We also show that the framework predicts the emergence of a conservative strategy that makes individuals to be ``cautious'' when interacting with ``acquaintances''.},
    Author = {Sequeira, Pedro and Mascarenhas, Samuel and Melo, Francisco S. and Paiva, Ana},
    Booktitle = {Proceedings of the 5th Joint International Conference on Development and Learning and on Epigenetic Robotics},
    Date-Modified = {2017-05-09 21:52:57 +0000},
    Doi = {10.1109/DEVLRN.2015.7346163},
    Isbn = {978-1-4673-9320-1},
    Location = {Providence, Rhode Island},
    Month = {Aug},
    Pages = {308--313},
    Publisher = {IEEE},
    Series = {ICDL-EpiRob 2015},
    Title = {{The Development of Cooperation in Evolving Populations through Social Importance}},
    Year = {2015},
    Bdsk-Url-1 = {http://dx.doi.org/10.1109/DEVLRN.2015.7346163}}

  • P. Alves-Oliveira, P. Sequeira, E. D. Tullio, S. Petisca, C. Guerra, F. S. Melo, and A. Paiva, ““It’s amazing, we are all feeling it!” Emotional Climate as a Group-Level Emotional Expression in HRI,” in Artificial Intelligence for Human-Robot Interaction, AAAI Fall Symposium Series, 2015.

    Emotions are a key element in all human interactions. It is well documented that individual- and group-level interactions have different emotional expressions and humans are by nature extremely competent in perceiving, adapting and reacting to them. However, when developing social robots, emotions are not so easy to cope with. In this paper we introduce the concept of emotional climate applied to human-robot interaction (HRI) to define a group-level emotional expression at a given time. By doing so, we move one step further in developing a new tool that deals with group emotions within HRI.

    @inproceedings{alves-oliveira2015aihri,
    Abstract = {Emotions are a key element in all human interactions. It is well documented that individual- and group-level interactions have different emotional expressions and humans are by nature extremely competent in perceiving, adapting and reacting to them. However, when developing social robots, emotions are not so easy to cope with. In this paper we introduce the concept of emotional climate applied to human-robot interaction (HRI) to define a group-level emotional expression at a given time. By doing so, we move one step further in developing a new tool that deals with group emotions within HRI.},
    Author = {Alves-Oliveira, Patr{\'{i}}cia and Sequeira, Pedro and Tullio, Eugenio Di and Petisca, Sofia and Guerra, Carla and Melo, Francisco S and Paiva, Ana},
    Booktitle = {Artificial Intelligence for Human-Robot Interaction, AAAI Fall Symposium Series},
    Date-Modified = {2017-05-09 22:17:03 +0000},
    Series = {AI-HRI 2015},
    Title = {{``It's amazing, we are all feeling it!'' Emotional Climate as a Group-Level Emotional Expression in HRI}},
    Year = {2015}}

  • [PDF] M. Vala, G. Raimundo, P. Sequeira, P. Cuba, R. Prada, C. Martinho, and A. Paiva, “ION Framework–-A Simulation Environment for Worlds with Virtual Agents,” in Proceedings of the 9th International Conference on Intelligent Virtual Agents, Amsterdam, The Netherlands, 2009, p. 418–424.

    Agents cannot be decoupled from their environment. An agent perceives and acts in a world and the model of the world influences how the agent makes decisions. Most systems with virtual embodied agents simulate the environment within a specific realization engine such as the graphics engine. As a consequence, these agents are bound to a particular kind of environment which compromises their reusability across different applications. We propose the ION Framework, a framework for simulating virtual environments which separates the simulation environment from the realization engine. In doing so, it facilitates the integration and reuse of the several components of the system. The ION Framework was used to create several 3D virtual worlds populated with autonomous embodied agents that were tested with hundreds of users.

    @inproceedings{vala2009iva,
    Abstract = {Agents cannot be decoupled from their environment. An agent perceives and acts in a world and the model of the world influences how the agent makes decisions. Most systems with virtual embodied agents simulate the environment within a specific realization engine such as the graphics engine. As a consequence, these agents are bound to a particular kind of environment which compromises their reusability across different applications. We propose the ION Framework, a framework for simulating virtual environments which separates the simulation environment from the realization engine. In doing so, it facilitates the integration and reuse of the several components of the system. The ION Framework was used to create several 3D virtual worlds populated with autonomous embodied agents that were tested with hundreds of users.},
    Address = {Berlin, Heidelberg},
    Author = {Vala, Marco and Raimundo, Guilherme and Sequeira, Pedro and Cuba, Pedro and Prada, Rui and Martinho, Carlos and Paiva, Ana},
    Booktitle = {Proceedings of the 9th International Conference on Intelligent Virtual Agents},
    Date-Modified = {2017-05-09 21:52:03 +0000},
    Doi = {10.1007/978-3-642-04380-2_45},
    Editor = {Ruttkay, Zs{\'o}fia and Kipp, Michael and Nijholt, Anton and Vilhj{\'a}lmsson, Hannes H{\"o}gni},
    Isbn = {978-3-642-04380-2},
    Location = {Amsterdam, The Netherlands},
    Month = {Sep},
    Pages = {418--424},
    Publisher = {Springer Berlin Heidelberg},
    Series = {Lecture Notes in Computer Science},
    Title = {{ION Framework---A Simulation Environment for Worlds with Virtual Agents}},
    Volume = {5773},
    Year = {2009},
    Bdsk-Url-1 = {http://dx.doi.org/10.1007/978-3-642-04380-2_45}}

  • [PDF] P. Sequeira, F. S. Melo, and A. Paiva, “Emergence of Emotional Appraisal Signals in Reinforcement Learning Agents (JAAMAS Extended Abstract),” in Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems, Singapore, 2016, p. 466–467.

    The positive impact of emotions in decision-making has long been established in both natural and artificial agents. In the perspective of appraisal theories, emotions complement perceptual information, coloring our sensations and guiding our decision-making. However, when designing autonomous agents, is emotional appraisal the best complement to their perceptions? Mechanisms investigated in the affective sciences provide support for this hypothesis in biological agents. In this paper, we look for similar support in artificial systems. We adopt the intrinsically motivated reinforcement learning framework to investigate different sources of information that can guide decision-making in learning agents, and an evolutionary approach based on genetic programming to identify a small set of such sources that have the largest impact on the performance of the agent in different tasks, as measured by an external evaluation signal. We then show that these sources of information: (i) are applicable in a wider range of environments than those where the agents evolved; (ii) exhibit interesting correspondences to emotional appraisal-like signals previously proposed in the literature. The results of the study thus point towards our departing hypothesis that the appraisal process might indeed provide essential information to complement perceptual capabilities and thus guide decision-making.

    @inproceedings{sequeira2016aamas,
    Abstract = {The positive impact of emotions in decision-making has long been established in both natural and artificial agents. In the perspective of appraisal theories, emotions complement perceptual information, coloring our sensations and guiding our decision-making. However, when designing autonomous agents, is emotional appraisal the best complement to their perceptions? Mechanisms investigated in the affective sciences provide support for this hypothesis in biological agents. In this paper, we look for similar support in artificial systems. We adopt the intrinsically motivated reinforcement learning framework to investigate different sources of information that can guide decision-making in learning agents, and an evolutionary approach based on genetic programming to identify a small set of such sources that have the largest impact on the performance of the agent in different tasks, as measured by an external evaluation signal. We then show that these sources of information: (i) are applicable in a wider range of environments than those where the agents evolved; (ii) exhibit interesting correspondences to emotional appraisal-like signals previously proposed in the literature. The results of the study thus point towards our departing hypothesis that the appraisal process might indeed provide essential information to complement perceptual capabilities and thus guide decision-making.},
    Address = {Richland, SC},
    Author = {Sequeira, Pedro and Melo, Francisco S and Paiva, Ana},
    Booktitle = {Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems},
    Date-Modified = {2017-05-03 21:23:49 +0000},
    Isbn = {978-1-4503-4239-1},
    Keywords = {appraisal theories, emotions, genetic programming, intrinsic motivation, reinforcement learning},
    Location = {Singapore},
    Month = {May},
    Pages = {466--467},
    Publisher = {International Foundation for Autonomous Agents and Multiagent Systems},
    Series = {AAMAS '16},
    Title = {{Emergence of Emotional Appraisal Signals in Reinforcement Learning Agents (JAAMAS Extended Abstract)}},
    Url = {http://dl.acm.org/citation.cfm?id=2936924.2936992},
    Year = {2016},
    Bdsk-Url-1 = {http://dl.acm.org/citation.cfm?id=2936924.2936992}}

  • [PDF] P. Sequeira and A. Paiva, “Learning to Interact: Connecting Perception with action in Virtual Environments,” in Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, Estoril, Portugal, 2008, p. 1257–1260.

    Modeling synthetic characters which interact with objects in dynamic virtual worlds is important when we want the agents to act in an autonomous and non-preplanned way. Such interactions with objects would allow the synthetic characters to behave in a more believable way. Once objects offer innumerous uses, it is essential that the agent is able to acquire the necessary knowledge to identify action possibilities in the objects while interacting with them. We propose a conceptual framework that allows the agents to identify possible interactions with objects based in past experiences with other objects. Starting from sensory patterns collected during interactions with objects, the agent is able to acquire conceptual knowledge about regularities of the world, its internal states and its own actions. The presented work also proposes that such acquired knowledge may be used by the agent in order to satisfy its needs and goals by interacting with objects. Preliminary tests were made and it is possible to state that our agents are able to acquire valid conceptual knowledge about the regularities in the environment and its objects, its own actions and causal relations between them.

    @inproceedings{sequeira2008aamas,
    Abstract = {Modeling synthetic characters which interact with objects in dynamic virtual worlds is important when we want the agents to act in an autonomous and non-preplanned way. Such interactions with objects would allow the synthetic characters to behave in a more believable way. Once objects offer innumerous uses, it is essential that the agent is able to acquire the necessary knowledge to identify action possibilities in the objects while interacting with them. We propose a conceptual framework that allows the agents to identify possible interactions with objects based in past experiences with other objects. Starting from sensory patterns collected during interactions with objects, the agent is able to acquire conceptual knowledge about regularities of the world, its internal states and its own actions. The presented work also proposes that such acquired knowledge may be used by the agent in order to satisfy its needs and goals by interacting with objects. Preliminary tests were made and it is possible to state that our agents are able to acquire valid conceptual knowledge about the regularities in the environment and its objects, its own actions and causal relations between them.},
    Address = {Richland, SC},
    Author = {Sequeira, Pedro and Paiva, Ana},
    Booktitle = {Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems},
    Date-Modified = {2017-05-09 22:24:25 +0000},
    Isbn = {978-0-9817381-2-3},
    Keywords = {Learning agents, believable qualities, object interaction, affordances, perception},
    Location = {Estoril, Portugal},
    Pages = {1257--1260},
    Publisher = {International Foundation for Autonomous Agents and Multiagent Systems},
    Series = {AAMAS '08},
    Title = {{Learning to Interact: Connecting Perception with action in Virtual Environments}},
    Year = {2008}}

  • [PDF] P. Sequeira and C. Antunes, “Real-Time Sensory Pattern Mining for Autonomous Agents,” in 6th International Workshop on Agents and Data Mining Interaction, ADMI 2010, Toronto, ON, Canada, 2010, p. 71–83.

    Autonomous agents are systems situated in dynamic environments. They pursue goals and satisfy their needs by responding to external events from the environment. In these unpredictable conditions, the agents’ adaptive skills are a key factor for their success. Based on previous interactions with its environment, an agent must learn new knowledge about it, and use that information to guide its behavior throughout time. In order to build more believable agents, we need to provide them with structures that represent that knowledge, and mechanisms that update them overtime to reflect the agents’ experience. Pattern mining, a subfield of data mining, is a knowledge discovery technique which aims to extract previously unknown associations and causal structures from existing data sources. In this paper we propose the use of pattern mining techniques in autonomous agents to allow the extraction of sensory patterns from the agent’s perceptions in real-time. We extend some structures used in pattern mining and employ a statistical test to allow an agent of discovering useful information about the environment while exploring it.

    @inproceedings{sequeira2010admi,
    Abstract = {Autonomous agents are systems situated in dynamic environments. They pursue goals and satisfy their needs by responding to external events from the environment. In these unpredictable conditions, the agents' adaptive skills are a key factor for their success. Based on previous interactions with its environment, an agent must learn new knowledge about it, and use that information to guide its behavior throughout time. In order to build more believable agents, we need to provide them with structures that represent that knowledge, and mechanisms that update them overtime to reflect the agents' experience. Pattern mining, a subfield of data mining, is a knowledge discovery technique which aims to extract previously unknown associations and causal structures from existing data sources. In this paper we propose the use of pattern mining techniques in autonomous agents to allow the extraction of sensory patterns from the agent's perceptions in real-time. We extend some structures used in pattern mining and employ a statistical test to allow an agent of discovering useful information about the environment while exploring it.},
    Address = {Berlin, Heidelberg},
    Author = {Sequeira, Pedro and Antunes, Cl{\'{a}}udia},
    Booktitle = {6th International Workshop on Agents and Data Mining Interaction, ADMI 2010},
    Date-Modified = {2017-05-09 22:24:44 +0000},
    Doi = {10.1007/978-3-642-15420-1_7},
    Editor = {Cao, Longbing and Bazzan, Ana L. C. and Gorodetsky, Vladimir and Mitkas, Pericles A. and Weiss, Gerhard and Yu, Philip S.},
    Isbn = {978-3-642-15420-1},
    Keywords = {Autonomous agents, adaptation, learning, pattern mining, knowledge discovery},
    Location = {Toronto, ON, Canada},
    Pages = {71--83},
    Publisher = {Springer Berlin Heidelberg},
    Series = {ADMI 2010},
    Title = {{Real-Time Sensory Pattern Mining for Autonomous Agents}},
    Year = {2010},
    Bdsk-Url-1 = {http://dx.doi.org/10.1007/978-3-642-15420-1_7}}

  • [PDF] P. Sequeira, F. S. Melo, and A. Paiva, “Emotion-Based Intrinsic Motivation for Reinforcement Learning Agents,” in 4th International Conference on Affective Computing and Intelligent Interaction, Memphis, TN, USA, 2011, p. 326–336. Best Paper Award.

    In this paper, we propose an adaptation of four common appraisal dimensions that evaluate the relation of an agent with its environment into reward features within an intrinsically motivated reinforcement learning framework. We show that, by optimizing the relative weights of such features for a given environment, the agents attain a greater degree of fitness while overcoming some of their perceptual limitations. This optimization process resembles the evolutionary adaptive process that living organisms are subject to. We illustrate the application of our method in several simulated foraging scenarios.

    @inproceedings{sequeira2011acii,
    Abstract = {In this paper, we propose an adaptation of four common appraisal dimensions that evaluate the relation of an agent with its environment into reward features within an intrinsically motivated reinforcement learning framework. We show that, by optimizing the relative weights of such features for a given environment, the agents attain a greater degree of fitness while overcoming some of their perceptual limitations. This optimization process resembles the evolutionary adaptive process that living organisms are subject to. We illustrate the application of our method in several simulated foraging scenarios.},
    Address = {Berlin, Heidelberg},
    Author = {Sequeira, Pedro and Melo, Francisco S. and Paiva, Ana},
    Award = {Best Paper Award},
    Booktitle = {4th International Conference on Affective Computing and Intelligent Interaction},
    Date-Modified = {2018-01-03 15:27:39 +0000},
    Doi = {10.1007/978-3-642-24600-5_36},
    Editor = {D'Mello, Sidney and Graesser, Arthur and Schuller, Bj{\"o}rn and Martin, Jean-Claude},
    Isbn = {978-3-642-24600-5},
    Keywords = {reinforcement learning, intrinsic motivation, appraisal},
    Location = {Memphis, TN, USA},
    Month = {Oct},
    Pages = {326--336},
    Publisher = {Springer Berlin Heidelberg},
    Series = {ACII 2011},
    Title = {{Emotion-Based Intrinsic Motivation for Reinforcement Learning Agents}},
    Volume = {6974},
    Year = {2011},
    Bdsk-Url-1 = {http://dx.doi.org/10.1007/978-3-642-24600-5_36}}

  • [PDF] R. Doroudi, R. Azghandi, Z. Feric, O. Mohaddesi, P. Sequeira, Y. Sun, J. Griffin, O. Ergun, C. Harteveld, D. Kaeli, and S. Marsella, “An Integrated Simulation Framework for Examining Resiliency in Pharmaceutical Supply Chains Considering Human Behaviors,” in Proceedings of the 2018 Winter Simulation Conference, 2018, p. to appear.

    @Conference{doroudi2018wsc,
    author = {Rozhin Doroudi and Rana Azghandi and Zlatan Feric and Omid Mohaddesi and Pedro Sequeira and Yifan Sun and Jacqueline Griffin and Ozlem Ergun and Casper Harteveld and David Kaeli and Stacy Marsella},
    title = {{An Integrated Simulation Framework for Examining Resiliency in Pharmaceutical Supply Chains Considering Human Behaviors}},
    booktitle = {Proceedings of the 2018 Winter Simulation Conference},
    year = {2018},
    series = {WSC},
    pages = {to appear},
    }

  • [PDF] P. Sequeira, S. Mascarenhas, F. S. Melo, and A. Paiva, “The “ Favors Game ”: A Framework to Study the Emergence of Cooperation through Social Importance,” in Proceedings of the 14th International Joint Conference on Autonomous Agents and Multiagent Systems, Istanbul, Turkey, 2015, p. 1899–1900.

    Several agent-based frameworks have been proposed to investigate the possible reasons that lead humans to act in the interest of others while giving up individual gains. In this paper we propose a novel framework for analyzing this phenomenon based on the notions of social importance and local discrimination. We propose a “Favors Game”, where a recipient agent can “claim” a favor to a donor agent, which may in turn “confer” its request at the expense of a certain cost. The proposed framework allows us to study the conditions under which cooperation occurs and the dynamics of the relationships formed within a population.

    @inproceedings{sequeira2015aamas,
    Abstract = {Several agent-based frameworks have been proposed to investigate the possible reasons that lead humans to act in the interest of others while giving up individual gains. In this paper we propose a novel framework for analyzing this phenomenon based on the notions of social importance and local discrimination. We propose a ``Favors Game'', where a recipient agent can ``claim'' a favor to a donor agent, which may in turn ``confer'' its request at the expense of a certain cost. The proposed framework allows us to study the conditions under which cooperation occurs and the dynamics of the relationships formed within a population.},
    Address = {Richland, SC},
    Author = {Sequeira, Pedro and Mascarenhas, Samuel and Melo, Francisco S. and Paiva, Ana},
    Booktitle = {Proceedings of the 14th International Joint Conference on Autonomous Agents and Multiagent Systems},
    Date-Modified = {2017-05-09 22:27:04 +0000},
    Isbn = {978-1-4503-3413-6},
    Keywords = {social importance, favors game, cooperation, social agents},
    Location = {Istanbul, Turkey},
    Month = {May},
    Pages = {1899--1900},
    Publisher = {International Foundation for Autonomous Agents and Multiagent Systems},
    Series = {AAMAS '15},
    Title = {{The `` Favors Game '': A Framework to Study the Emergence of Cooperation through Social Importance}},
    Year = {2015}}

  • [PDF] P. Sequeira, M. Vala, and A. Paiva, “What Can I Do With This?: Finding Possible Interactions Between Characters And Objects,” in Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems, Honolulu, Hawaii, 2007, p. 5:1–5:7.

    Virtual environments are often populated by autonomous synthetic agents capable of acting and interacting with other agents as well as with humans. These virtual worlds also include objects that may have different uses and types of interactions. As such, these agents need to identify possible interactions with the objects in the environment and measure the consequences of these interactions. This is particularly difficult when the agents never interacted with some of the objects beforehand. This paper describes SOTAI – Smart ObjecT-Agent Interaction, a framework that will help agents to identify possible interactions with unknown objects based on their past experiences. In SOTAI, agents can learn world regularities, like object attributes and frequent relations between attributes. They gather qualitative symbolic descriptions from their sensorial data when interacting with objects and perform inductive reasoning to acquire concepts about them. We implemented an initial case study and the results show that our agents are able to acquire valid conceptual knowledge.

    @inproceedings{sequeira2007aamas,
    Abstract = {Virtual environments are often populated by autonomous synthetic agents capable of acting and interacting with other agents as well as with humans. These virtual worlds also include objects that may have different uses and types of interactions. As such, these agents need to identify possible interactions with the objects in the environment and measure the consequences of these interactions. This is particularly difficult when the agents never interacted with some of the objects beforehand. This paper describes SOTAI - Smart ObjecT-Agent Interaction, a framework that will help agents to identify possible interactions with unknown objects based on their past experiences. In SOTAI, agents can learn world regularities, like object attributes and frequent relations between attributes. They gather qualitative symbolic descriptions from their sensorial data when interacting with objects and perform inductive reasoning to acquire concepts about them. We implemented an initial case study and the results show that our agents are able to acquire valid conceptual knowledge.},
    Address = {USA},
    Author = {Sequeira, Pedro and Vala, Marco and Paiva, Ana},
    Booktitle = {Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems},
    Date-Modified = {2017-05-09 22:24:04 +0000},
    Doi = {10.1145/1329125.1329132},
    Isbn = {978-81-904262-7-5},
    Keywords = {synthetic agents: human-like, lifelike, believable qualities, learning agents},
    Location = {Honolulu, Hawaii},
    Pages = {5:1--5:7},
    Publisher = {ACM},
    Series = {AAMAS '07},
    Title = {{What Can I Do With This?: Finding Possible Interactions Between Characters And Objects}},
    Year = {2007},
    Bdsk-Url-1 = {http://dx.doi.org/10.1145/1329125.1329132}}

  • [PDF] P. Sequeira, P. Alves-Oliveira, T. Ribeiro, E. D. Tullio, S. Petisca, F. S. Melo, G. Castellano, and A. Paiva, “Discovering Social Interaction Strategies for Robots from Restricted-Perception Wizard-of-Oz Studies,” in Proceedings of the 11th ACM/IEEE International Conference on Human-Robot Interaction, Christchurch, New Zealand, 2016, p. 197–204. Best Paper Award on HRI Interaction Design.

    In this paper we propose a methodology for the creation of social interaction strategies for human-robot interaction based on restricted-perception Wizard-of-Oz studies (WoZ). This novel experimental technique involves restricting the wizard’s perceptions over the environment and the behaviors it controls according to the robot’s inherent perceptual and acting limitations. Within our methodology, the robot’s design lifecycle is divided into three consecutive phases, namely data collection, where we perform interaction studies to extract expert knowledge and interaction data; strategy extraction, where a hybrid strategy controller for the robot is learned based on the gathered data; strategy refinement, where the controller is iteratively evaluated and adjusted. We developed a fully-autonomous robotic tutor based on the proposed approach in the context of a collaborative learning scenario. The results of the evaluation study show that, by performing restricted-perception WoZ studies, our robots are able to engage in very natural and socially-aware interactions.

    @inproceedings{sequeira2016hri,
    Abstract = {In this paper we propose a methodology for the creation of social interaction strategies for human-robot interaction based on restricted-perception Wizard-of-Oz studies (WoZ). This novel experimental technique involves restricting the wizard's perceptions over the environment and the behaviors it controls according to the robot's inherent perceptual and acting limitations. Within our methodology, the robot's design lifecycle is divided into three consecutive phases, namely data collection, where we perform interaction studies to extract expert knowledge and interaction data; strategy extraction, where a hybrid strategy controller for the robot is learned based on the gathered data; strategy refinement, where the controller is iteratively evaluated and adjusted. We developed a fully-autonomous robotic tutor based on the proposed approach in the context of a collaborative learning scenario. The results of the evaluation study show that, by performing restricted-perception WoZ studies, our robots are able to engage in very natural and socially-aware interactions.},
    Author = {Sequeira, Pedro and Alves-Oliveira, Patr{\'{i}}cia and Ribeiro, Tiago and Tullio, Eugenio Di and Petisca, Sofia and Melo, Francisco S. and Castellano, Ginevra and Paiva, Ana},
    Award = {Best Paper Award on HRI Interaction Design},
    Booktitle = {Proceedings of the 11th ACM/IEEE International Conference on Human-Robot Interaction},
    Date-Modified = {2018-01-03 15:27:14 +0000},
    Doi = {10.1109/HRI.2016.7451752},
    Isbn = {978-1-4673-8370-7},
    Location = {Christchurch, New Zealand},
    Month = {Mar},
    Pages = {197--204},
    Publisher = {IEEE},
    Series = {HRI 2016},
    Title = {{Discovering Social Interaction Strategies for Robots from Restricted-Perception Wizard-of-Oz Studies}},
    Year = {2016},
    Bdsk-Url-1 = {http://dx.doi.org/10.1109/HRI.2016.7451752}}

  • S. Chandra, P. Alves-Oliveira, S. Lemaignan, P. Sequeira, A. Paiva, and P. Dillenbourg, “Can a Child Feel Responsible for Another in the Presence of a Robot in a Collaborative Learning Activity ?,” in Proceedings of the 24th IEEE International Symposium on Robot and Human Interactive Communication, 2015, p. 167–172.

    In order to explore the impact of integrating a robot as a facilitator in a collaborative activity, we examined interpersonal distancing of children both with a human adult and a robot facilitator. Our scenario involves two children performing a collaborative learning activity, which included the writing of a word/letter on a tactile tablet. Based on the learning-by-teaching paradigm, one of the children acted as a teacher when the other acted as a learner. Our study involved 40 children between 6 and 8 years old, in two conditions (robot or human facilitator). The results suggest first that the child acting as a teacher feel more responsible when the facilitator is a robot, compared to a human; they show then that the interaction between a (teacher) child and a robot facilitator can be characterized as being a reciprocity-based interaction, whereas a human presence fosters a compensation-based interaction.

    @inproceedings{chandra2015roman,
    Abstract = {In order to explore the impact of integrating a robot as a facilitator in a collaborative activity, we examined interpersonal distancing of children both with a human adult and a robot facilitator. Our scenario involves two children performing a collaborative learning activity, which included the writing of a word/letter on a tactile tablet. Based on the learning-by-teaching paradigm, one of the children acted as a teacher when the other acted as a learner. Our study involved 40 children between 6 and 8 years old, in two conditions (robot or human facilitator). The results suggest first that the child acting as a teacher feel more responsible when the facilitator is a robot, compared to a human; they show then that the interaction between a (teacher) child and a robot facilitator can be characterized as being a reciprocity-based interaction, whereas a human presence fosters a compensation-based interaction.},
    Author = {Chandra, Shruti and Alves-Oliveira, Patr{\'{i}}cia and Lemaignan, S{\'{e}}verin and Sequeira, Pedro and Paiva, Ana and Dillenbourg, Pierre},
    Booktitle = {Proceedings of the 24th IEEE International Symposium on Robot and Human Interactive Communication},
    Date-Modified = {2017-05-09 21:56:48 +0000},
    Doi = {10.1109/ROMAN.2015.7333678},
    Isbn = {978-1-4673-6704-2},
    Pages = {167--172},
    Publisher = {IEEE},
    Series = {RO-MAN 2015},
    Title = {{Can a Child Feel Responsible for Another in the Presence of a Robot in a Collaborative Learning Activity ?}},
    Year = {2015},
    Bdsk-Url-1 = {http://dx.doi.org/10.1109/ROMAN.2015.7333678}}

  • [PDF] M. Vala, P. Sequeira, A. Paiva, and R. Aylett, “FearNot! demo: a virtual environment with synthetic characters to help bullying,” in Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems, Honolulu, Hawaii, 2007, p. 271:1–271:2. Best Demo Award.

    This demo features FearNot!, a school-based Virtual Learning Environment (VLE) populated by synthetic characters representing the various actors in a bullying scenario. FearNot! uses emergent narrative to create improvised dramas with those characters. The goal is to enable children to explore bullying issues, and coping strategies, interacting with characters to which they become affectively engaged. Through their appearance, behaviours and affect, these characters are able to trigger empathic relations with the user. FearNot! is used for Personal and Health Social Education (PHSE) for children aged 8–12, in the UK, Portugal and Germany.

    @inproceedings{vala2007aamas,
    Abstract = {This demo features FearNot!, a school-based Virtual Learning Environment (VLE) populated by synthetic characters representing the various actors in a bullying scenario. FearNot! uses emergent narrative to create improvised dramas with those characters. The goal is to enable children to explore bullying issues, and coping strategies, interacting with characters to which they become affectively engaged. Through their appearance, behaviours and affect, these characters are able to trigger empathic relations with the user. FearNot! is used for Personal and Health Social Education (PHSE) for children aged 8--12, in the UK, Portugal and Germany.},
    Address = {New York, NY, USA},
    Author = {Vala, M. and Sequeira, P. and Paiva, A. and Aylett, R.},
    Award = {Best Demo Award},
    Booktitle = {{Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems}},
    Date-Modified = {2018-01-03 15:28:17 +0000},
    Doi = {10.1145/1329125.1329452},
    Isbn = {978-81-904262-7-5},
    Location = {Honolulu, Hawaii},
    Month = {May},
    Pages = {271:1--271:2},
    Publisher = {ACM},
    Series = {AAMAS '07},
    Title = {{FearNot! demo: a virtual environment with synthetic characters to help bullying}},
    Year = {2007},
    Bdsk-Url-1 = {http://dx.doi.org/10.1145/1329125.1329452}}

  • T. Ribeiro, P. Alves-Oliveira, E. Di Tullio, S. Petisca, P. Sequeira, A. Deshmukh, S. Janarthanam, M. E. Foster, A. Jones, L. J. Corrigan, F. Papadopoulos, H. Hastie, R. Aylett, G. Castellano, and A. Paiva, “The Empathic Robotic Tutor: Featuring the NAO Robot,” in Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts, Portland, Oregon, USA, 2015, p. 285–285.

    We present an autonomous empathic robotic tutor to be used in classrooms as a peer in a virtual learning environment. The system merges a virtual agent design with HRI features, consisting of a robotic embodiment, a multimedia interactive learning application and perception sensors that are controlled by an artificial intelligence agent.

    @inproceedings{ribeiro2015hri,
    Abstract = {We present an autonomous empathic robotic tutor to be used in classrooms as a peer in a virtual learning environment. The system merges a virtual agent design with HRI features, consisting of a robotic embodiment, a multimedia interactive learning application and perception sensors that are controlled by an artificial intelligence agent.},
    Address = {New York, NY, USA},
    Author = {Ribeiro, Tiago and Alves-Oliveira, Patr\'{\i}cia and Di Tullio, Eugenio and Petisca, Sofia and Sequeira, Pedro and Deshmukh, Amol and Janarthanam, Srinivasan and Foster, Mary Ellen and Jones, Aidan and Corrigan, Lee J. and Papadopoulos, Fotios and Hastie, Helen and Aylett, Ruth and Castellano, Ginevra and Paiva, Ana},
    Booktitle = {Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts},
    Date-Added = {2017-05-09 22:21:32 +0000},
    Date-Modified = {2017-05-09 22:30:09 +0000},
    Doi = {10.1145/2701973.2702100},
    Isbn = {978-1-4503-3318-4},
    Keywords = {educational robotics, empathic robot},
    Location = {Portland, Oregon, USA},
    Pages = {285--285},
    Publisher = {ACM},
    Series = {HRI'15 Extended Abstracts},
    Title = {The Empathic Robotic Tutor: Featuring the NAO Robot},
    Url = {http://doi.acm.org/10.1145/2701973.2702100},
    Year = {2015},
    Bdsk-Url-1 = {http://doi.acm.org/10.1145/2701973.2702100},
    Bdsk-Url-2 = {http://dx.doi.org/10.1145/2701973.2702100}}

  • S. Chandra, P. Alves-Oliveira, S. Lemaignan, P. Sequeira, A. Paiva, and P. Dillenbourg, “Children’s peer assessment and self-disclosure in the presence of an educational robot,” in 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2016, pp. 539-544.

    Research in education has long established how children mutually influence and support each other’s learning trajectories, eventually leading to the development and widespread use of learning methods based on peer activities. In order to explore children’s learning behavior in the presence of a robotic facilitator during a collaborative writing activity, we investigated how they assess their peers in two specific group learning situations: peer-tutoring and peer-learning. Our scenario comprises of a pair of children performing a collaborative activity involving the act of writing a word/letter on a tactile tablet. In the peer-tutoring condition, one child acts as the teacher and the other as the learner, while in the peer-learning condition, both children are learners without the attribution of any specific role. Our experiment includes 40 children in total (between 6 and 8 years old) over the two conditions, each time in the presence of a robot facilitator. Our results suggest that the peer-tutoring situation leads to significantly more corrective feedback being provided, as well as the children more disposed to self-disclosure to the robot.

    @inproceedings{chandra2016roman,
    Abstract = {Research in education has long established how children mutually influence and support each other's learning trajectories, eventually leading to the development and widespread use of learning methods based on peer activities. In order to explore children's learning behavior in the presence of a robotic facilitator during a collaborative writing activity, we investigated how they assess their peers in two specific group learning situations: peer-tutoring and peer-learning. Our scenario comprises of a pair of children performing a collaborative activity involving the act of writing a word/letter on a tactile tablet. In the peer-tutoring condition, one child acts as the teacher and the other as the learner, while in the peer-learning condition, both children are learners without the attribution of any specific role. Our experiment includes 40 children in total (between 6 and 8 years old) over the two conditions, each time in the presence of a robot facilitator. Our results suggest that the peer-tutoring situation leads to significantly more corrective feedback being provided, as well as the children more disposed to self-disclosure to the robot.},
    Author = {S. Chandra and P. Alves-Oliveira and S. Lemaignan and P. Sequeira and A. Paiva and P. Dillenbourg},
    Booktitle = {2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)},
    Date-Added = {2017-05-09 22:20:39 +0000},
    Date-Modified = {2017-05-09 22:20:58 +0000},
    Doi = {10.1109/ROMAN.2016.7745170},
    Keywords = {behavioural sciences computing;computer aided instruction;educational robots;children learning behavior;children peer assessment;children performing;collaborative writing activity;educational robot;group learning situations;learning methods;peer activities;peer-learning;peer-learning condition;peer-tutoring;peer-tutoring condition;tactile tablet;Collaboration;Collaborative work;Context;Electronic mail;Learning systems;Robots;Writing},
    Month = {Aug},
    Pages = {539-544},
    Title = {Children's peer assessment and self-disclosure in the presence of an educational robot},
    Year = {2016},
    Bdsk-Url-1 = {http://dx.doi.org/10.1109/ROMAN.2016.7745170}}

  • [PDF] P. Sequeira, “Emergence of Emotional Appraisal Signals in Reinforcement Learning Agents,” in Pre-Conference on Affective Computing @ the 2017 Society for Affective Science Annual Conference, 2017.

    The positive impact of emotions in decision-making has long been established in both natural and artificial agents. In the perspective of appraisal theories, emotions complement perceptual information, coloring our sensations and guiding our decision-making. However, when designing autonomous agents, is emotional appraisal the best complement to their perceptions? Mechanisms investigated in the affective sciences provide support for this hypothesis in biological agents. In this paper, we look for similar support in artificial systems. We adopt the intrinsically motivated reinforcement learning framework to investigate dif- ferent sources of information that can guide decision-making in learning agents, and an evolutionary approach based on genetic programming to identify a small set of such sources that have the largest impact on the performance of the agent in different tasks, as measured by an external evaluation signal. We then show that these sources of information: (i) are applicable in a wider range of environments than those where the agents evolved; (ii) exhibit interesting correspondences to emotional appraisal- like signals previously proposed in the literature. The results of the study thus point towards our departing hypothesis that the appraisal process might indeed provide essential information to complement perceptual capabilities and thus guide decision-making.

    @conference{sequeira2017sas,
    Abstract = {The positive impact of emotions in decision-making has long been established in both natural and artificial agents. In the perspective of appraisal theories, emotions complement perceptual information, coloring our sensations and guiding our decision-making. However, when designing autonomous agents, is emotional appraisal the best complement to their perceptions? Mechanisms investigated in the affective sciences provide support for this hypothesis in biological agents. In this paper, we look for similar support in artificial systems. We adopt the intrinsically motivated reinforcement learning framework to investigate dif- ferent sources of information that can guide decision-making in learning agents, and an evolutionary approach based on genetic programming to identify a small set of such sources that have the largest impact on the performance of the agent in different tasks, as measured by an external evaluation signal. We then show that these sources of information: (i) are applicable in a wider range of environments than those where the agents evolved; (ii) exhibit interesting correspondences to emotional appraisal- like signals previously proposed in the literature. The results of the study thus point towards our departing hypothesis that the appraisal process might indeed provide essential information to complement perceptual capabilities and thus guide decision-making.},
    Author = {Pedro Sequeira},
    Booktitle = {Pre-Conference on Affective Computing @ the 2017 Society for Affective Science Annual Conference},
    Date-Added = {2017-05-09 22:35:46 +0000},
    Date-Modified = {2017-05-09 22:46:59 +0000},
    Month = {April},
    Title = {Emergence of Emotional Appraisal Signals in Reinforcement Learning Agents},
    Year = {2017}}

  • P. Sequeira, “Creating more Autonomous, Flexible and Robust Artificial Agents inspired by Nature,” in CNC’16: SIGAI Career Network Conference, 2016.

    I am currently an associate research scientist at Northeastern University, Cognitive Embodied Social Agents Research Lab (CESAR). Before that I was a post-doctoral associate at the Intelligent Agents and Synthetic Characters Group (GAIPS) / INESC-ID Lisbon in Portugal. In September 2013 I completed the Ph.D. Program in Information Systems and Computer Engineering at the Instituto Superior Técnico (IST), Universidade de Lisboa in Portugal. My thesis focused on building more flexible and robust reward mechanisms for autonomous reinforcement learning (RL) agents. My interests are in the area of artificial intelligence, particularly related with autonomous and adaptive systems (e.g. agents and multiagent systems) involving learning and evolutionary mechanisms. My approach has been greatly inspired by adaptive biological mechanisms.

    @conference{sequeira2016cnc,
    Abstract = {I am currently an associate research scientist at Northeastern University, Cognitive Embodied Social Agents Research Lab (CESAR). Before that I was a post-doctoral associate at the Intelligent Agents and Synthetic Characters Group (GAIPS) / INESC-ID Lisbon in Portugal. In September 2013 I completed the Ph.D. Program in Information Systems and Computer Engineering at the Instituto Superior T{\'e}cnico (IST), Universidade de Lisboa in Portugal. My thesis focused on building more flexible and robust reward mechanisms for autonomous reinforcement learning (RL) agents. My interests are in the area of artificial intelligence, particularly related with autonomous and adaptive systems (e.g. agents and multiagent systems) involving learning and evolutionary mechanisms. My approach has been greatly inspired by adaptive biological mechanisms.},
    Author = {Pedro Sequeira},
    Booktitle = {CNC'16: SIGAI Career Network Conference},
    Date-Added = {2017-05-09 22:42:52 +0000},
    Date-Modified = {2017-05-09 22:47:21 +0000},
    Month = {October},
    Title = {Creating more Autonomous, Flexible and Robust Artificial Agents inspired by Nature},
    Year = {2016}}

  • [PDF] Y. Xu, P. Sequeira, and S. Marsella, “Towards Modeling Agent Negotiators by Analyzing Human Negotiation Behavior,” in 7th International Conference on Affective Computing and Intelligent Interaction, 2017, pp. 58-64.

    @inproceedings{xu2017acii,
    Address = {San Antonio, TX, USA},
    Author = {Yuyu Xu and Pedro Sequeira and Stacy Marsella},
    Booktitle = {7th International Conference on Affective Computing and Intelligent Interaction},
    Date-Added = {2018-04-19 23:37:17 +0000},
    Date-Modified = {2018-04-19 23:40:19 +0000},
    Doi = {10.1109/ACII.2017.8273579},
    Keywords = {behavioural sciences computing;learning (artificial intelligence);multi-agent systems;software agents;agent player;artificial agent negotiators;data-driven approach;fixed-response strategy;human behavior patterns;human negotiation behavior;human negotiation skills;human negotiation styles;human negotiation trainees;machine learning techniques;personality traits;social interaction;Computational modeling;Cultural differences;Data collection;Data models;Games;Task analysis;Training},
    Pages = {58-64},
    Series = {ACII 2017},
    Title = {{Towards Modeling Agent Negotiators by Analyzing Human Negotiation Behavior}},
    Year = {2017},
    Bdsk-Url-1 = {https://doi.org/10.1109/ACII.2017.8273579}}

  • P. Alves-Oliveira, P. Sequeira, and A. Paiva, “The Role that an Educational Robot Plays,” in 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2016, pp. 817-822.

    Human beings naturally assign roles to one another while interacting. Role assignment is a way to organize interpersonal encounters and can result in uncertainty decrease when facing a novel interaction with someone we just met, or even to rediscover new roles within previous relationships. When people interact with synthetic characters – such as robots – it seems they also assign roles to these agents, just as they do with humans. Within the field of human-robot interaction (HRI), robots are being developed to fulfill specific roles. This enables researchers to design concrete behaviors that match the desired role that a robot will play in a given task. It would then be expected that if a robot is developed with such a specific role, users too would assign the same role to that robot. In this paper, we study how children assign roles to an educational robot whose role is established from the beginning of the interaction. Our results show that although the role that the robot played was explicitly presented to children, they end up perceiving and assigning different roles for that robot. Moreover, we conclude that role assignment in educational HRI is a dynamic process in which the perceptions of children regarding the robot change over time as a consequence of continuous interactions.

    @inproceedings{alves-oliveira2016roman,
    Abstract = {Human beings naturally assign roles to one another while interacting. Role assignment is a way to organize interpersonal encounters and can result in uncertainty decrease when facing a novel interaction with someone we just met, or even to rediscover new roles within previous relationships. When people interact with synthetic characters - such as robots - it seems they also assign roles to these agents, just as they do with humans. Within the field of human-robot interaction (HRI), robots are being developed to fulfill specific roles. This enables researchers to design concrete behaviors that match the desired role that a robot will play in a given task. It would then be expected that if a robot is developed with such a specific role, users too would assign the same role to that robot. In this paper, we study how children assign roles to an educational robot whose role is established from the beginning of the interaction. Our results show that although the role that the robot played was explicitly presented to children, they end up perceiving and assigning different roles for that robot. Moreover, we conclude that role assignment in educational HRI is a dynamic process in which the perceptions of children regarding the robot change over time as a consequence of continuous interactions.},
    Author = {P. Alves-Oliveira and P. Sequeira and A. Paiva},
    Booktitle = {2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)},
    Date-Added = {2017-05-09 22:18:07 +0000},
    Date-Modified = {2017-05-09 22:19:18 +0000},
    Doi = {10.1109/ROMAN.2016.7745213},
    Keywords = {educational robots;human-robot interaction;continuous interactions;educational HRI;educational robot;human-robot interaction;role assignment;synthetic characters;Context;Educational robots;Games;Human-robot interaction;Investment;Speech},
    Month = {Aug},
    Pages = {817-822},
    Title = {{The Role that an Educational Robot Plays}},
    Year = {2016},
    Bdsk-Url-1 = {http://dx.doi.org/10.1109/ROMAN.2016.7745213}}

  • [PDF] P. Sequeira, F. S. Melo, and A. Paiva, “The Influence of Social Display in Competitive Multiagent Learning,” in Proceedings of the 4th International Conference on Development and Learning and on Epigenetic Robotics, Genoa, Italy, 2014, p. 64–69.

    In this paper we analyze the impact of simple social signaling mechanisms in the performance of learning agents within competitive multiagent settings. In our framework, self-interested reinforcement learning agents interact and compete with each other for limited resources. The agents can exchange social signals that influence the total amount of reward received throughout time. In a series of experiments, we vary the amount resources available in the environment, the frequency of interactions and the importance each agent in the population gives to the social displays of others. We measure the combined performance of the population according to distinct selection paradigms based on the individual performances of each agent. The results of our study show that by focusing on the social displays of others, agents learn to collectively coordinate their feeding behavior by trading-off immediate benefit for long-term social welfare. Also, given populations where the impact of the social signal on the reward differs, the individuals with the highest fitness appear in the most socially-aware populations. The presence of social signaling gives also origin to more social inequalities where the more fit agents benefit from their higher status being appreciated by other members of the population.

    @inproceedings{sequeira2014icdl,
    Abstract = {In this paper we analyze the impact of simple social signaling mechanisms in the performance of learning agents within competitive multiagent settings. In our framework, self-interested reinforcement learning agents interact and compete with each other for limited resources. The agents can exchange social signals that influence the total amount of reward received throughout time. In a series of experiments, we vary the amount resources available in the environment, the frequency of interactions and the importance each agent in the population gives to the social displays of others. We measure the combined performance of the population according to distinct selection paradigms based on the individual performances of each agent. The results of our study show that by focusing on the social displays of others, agents learn to collectively coordinate their feeding behavior by trading-off immediate benefit for long-term social welfare. Also, given populations where the impact of the social signal on the reward differs, the individuals with the highest fitness appear in the most socially-aware populations. The presence of social signaling gives also origin to more social inequalities where the more fit agents benefit from their higher status being appreciated by other members of the population.},
    Author = {Sequeira, Pedro and Melo, Francisco S and Paiva, Ana},
    Booktitle = {Proceedings of the 4th International Conference on Development and Learning and on Epigenetic Robotics},
    Date-Modified = {2017-05-09 21:53:19 +0000},
    Doi = {10.1109/DEVLRN.2014.6982954},
    Isbn = {978-1-4799-7540-2},
    Location = {Genoa, Italy},
    Month = {Oct},
    Pages = {64--69},
    Publisher = {IEEE},
    Series = {ICDL-EpiRob 2014},
    Title = {{The Influence of Social Display in Competitive Multiagent Learning}},
    Year = {2014},
    Bdsk-Url-1 = {http://dx.doi.org/10.1109/DEVLRN.2014.6982954}}

  • [PDF] P. Sequeira, F. S. Melo, and A. Paiva, “An Associative State-Space Metric for Learning in Factored MDPs,” in Proceedings of the 16th Portuguese Conference on Artificial Intelligence, Angra do Heroísmo, Portugal, 2013, p. 163–174.

    In this paper we propose a novel associative metric based on the classical conditioning paradigm that, much like what happens in nature, identifies associations between stimuli perceived by a learning agent while interacting with the environment. We use an associative tree structure to identify associations between the perceived stimuli and use this structure to measure the degree of similarity between states in factored Markov decision problems. Our approach provides a state-space metric that requires no prior knowledge on the structure of the underlying decision problem and is designed to be learned online, i.e., as the agent interacts with its environment. Our metric is thus amenable to application in reinforcement learning (RL) settings, allowing the learning agent to generalize its experience to unvisited states and improving the overall learning performance. We illustrate the application of our method in several problems of varying complexity and show that our metric leads to a performance comparable to that obtained with other well-studied metrics that require full knowledge of the decision problem.

    @inproceedings{sequeira2013epia,
    Abstract = {In this paper we propose a novel associative metric based on the classical conditioning paradigm that, much like what happens in nature, identifies associations between stimuli perceived by a learning agent while interacting with the environment. We use an associative tree structure to identify associations between the perceived stimuli and use this structure to measure the degree of similarity between states in factored Markov decision problems. Our approach provides a state-space metric that requires no prior knowledge on the structure of the underlying decision problem and is designed to be learned online, i.e., as the agent interacts with its environment. Our metric is thus amenable to application in reinforcement learning (RL) settings, allowing the learning agent to generalize its experience to unvisited states and improving the overall learning performance. We illustrate the application of our method in several problems of varying complexity and show that our metric leads to a performance comparable to that obtained with other well-studied metrics that require full knowledge of the decision problem.},
    Address = {Berlin, Heidelberg},
    Author = {Sequeira, Pedro and Melo, Francisco S. and Paiva, Ana},
    Booktitle = {Proceedings of the 16th Portuguese Conference on Artificial Intelligence},
    Date-Modified = {2017-05-09 22:33:29 +0000},
    Doi = {10.1007/978-3-642-40669-0_15},
    Editor = {Correia, Lu{\'\i}s and Reis, Lu{\'\i}s Paulo and Cascalho, Jos{\'e}},
    Isbn = {978-3-642-40669-0},
    Location = {Angra do Hero{\'{i}}smo, Portugal},
    Month = {Sep},
    Pages = {163--174},
    Publisher = {Springer Berlin Heidelberg},
    Series = {EPIA 2013},
    Title = {{An Associative State-Space Metric for Learning in Factored MDPs}},
    Year = {2013},
    Bdsk-Url-1 = {http://dx.doi.org/10.1007/978-3-642-40669-0_15}}

  • [PDF] P. Sequeira, F. S. Melo, and A. Paiva, ““Let’s Save Resources!”: A Dynamic, Collaborative AI for a Multiplayer Environmental Awareness Game,” in Proceedings of the 2015 IEEE Conference on Computational Intelligence and Games, Tainan, Taiwan, 2015, p. 399–406.

    In this paper we present a collaborative artificial intelligence (AI) module for a turn-based, multiplayer, environmental awareness game. The game is a version of the EnerCities serious game, modified in the context of a European-Union project to support sequential plays of an emphatic robotic tutor interacting with two human players in a social and pedagogical manner. For that purpose, we created an AI module capable of informing the game-playing and pedagogical decision-making of the robotic tutor. Specifically, the module includes an action planner capable of, together with a game simulator, perform forward-planning according to player preferences and current game values. Such predicted values are also used as an alert system to inform the other players of near consequences of current behaviors and advise alternative, sustainable courses of action in the game. The module also incorporates a social component that continuously models the game preferences of each player and automatically adjusts the tutor’s strategy so to follow the group’s “action tendency”. The proposed AI module is therefore used to inform about important aspects of the game state and also the human players actions. In this paper we overview the properties and complexity of this collaborative version of the game and detail the AI module and its components. We also report on the successes of using the proposed module for controlling the behavior of a robotic tutor in several experimental studies, including the interaction with children playing collaborative EnerCities.

    @inproceedings{sequeira2015cig,
    Abstract = {In this paper we present a collaborative artificial intelligence (AI) module for a turn-based, multiplayer, environmental awareness game. The game is a version of the EnerCities serious game, modified in the context of a European-Union project to support sequential plays of an emphatic robotic tutor interacting with two human players in a social and pedagogical manner. For that purpose, we created an AI module capable of informing the game-playing and pedagogical decision-making of the robotic tutor. Specifically, the module includes an action planner capable of, together with a game simulator, perform forward-planning according to player preferences and current game values. Such predicted values are also used as an alert system to inform the other players of near consequences of current behaviors and advise alternative, sustainable courses of action in the game. The module also incorporates a social component that continuously models the game preferences of each player and automatically adjusts the tutor's strategy so to follow the group's ``action tendency''. The proposed AI module is therefore used to inform about important aspects of the game state and also the human players actions. In this paper we overview the properties and complexity of this collaborative version of the game and detail the AI module and its components. We also report on the successes of using the proposed module for controlling the behavior of a robotic tutor in several experimental studies, including the interaction with children playing collaborative EnerCities.},
    Author = {Sequeira, Pedro and Melo, Francisco S. and Paiva, Ana},
    Booktitle = {Proceedings of the 2015 IEEE Conference on Computational Intelligence and Games},
    Date-Modified = {2017-05-09 21:52:14 +0000},
    Doi = {10.1109/CIG.2015.7317952},
    Isbn = {978-1-4799-8622-4},
    Location = {Tainan, Taiwan},
    Month = {Aug},
    Pages = {399--406},
    Publisher = {IEEE},
    Series = {IEEE CIG 2015},
    Title = {{``Let's Save Resources!'': A Dynamic, Collaborative AI for a Multiplayer Environmental Awareness Game}},
    Year = {2015},
    Bdsk-Url-1 = {http://dx.doi.org/10.1109/CIG.2015.7317952}}

  • [PDF] P. Sequeira, F. S. Melo, R. Prada, and A. Paiva, “Emerging social awareness: Exploring intrinsic motivation in multiagent learning,” in Proceedings of the 1st Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics, Frankfurt, Germany, 2011, p. 1–6. Best Poster Award.

    Recently, a novel framework has been proposed for intrinsically motivated reinforcement learning (IMRL) in which a learning agent is driven by rewards that include not only information about what the agent must accomplish in order to “survive”, but also additional reward signals that drive the agent to engage in other activities, such as playing or exploring, because they are “inherently enjoyable”. In this paper, we investigate the impact of intrinsic motivation mechanisms in multiagent learning scenarios, by considering how such motivational system may drive an agent to engage in behaviors that are “socially aware”. We show that, using this approach, it is possible for agents to learn individually to acquire socially aware behaviors that tradeoff individual well-fare for social acknowledgment, leading to a more successful performance of the population as a whole.

    @inproceedings{sequeira2011icdl,
    Abstract = {Recently, a novel framework has been proposed for intrinsically motivated reinforcement learning (IMRL) in which a learning agent is driven by rewards that include not only information about what the agent must accomplish in order to ``survive'', but also additional reward signals that drive the agent to engage in other activities, such as playing or exploring, because they are ``inherently enjoyable''. In this paper, we investigate the impact of intrinsic motivation mechanisms in multiagent learning scenarios, by considering how such motivational system may drive an agent to engage in behaviors that are ``socially aware''. We show that, using this approach, it is possible for agents to learn individually to acquire socially aware behaviors that tradeoff individual well-fare for social acknowledgment, leading to a more successful performance of the population as a whole.},
    Author = {Sequeira, Pedro and Melo, Francisco S. and Prada, Rui and Paiva, Ana},
    Award = {Best Poster Award},
    Booktitle = {Proceedings of the 1st Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics},
    Date-Modified = {2018-01-03 15:27:59 +0000},
    Doi = {10.1109/DEVLRN.2011.6037325},
    Isbn = {978-1-61284-990-4},
    Location = {Frankfurt, Germany},
    Pages = {1--6},
    Publisher = {IEEE},
    Series = {ICDL-EpiRob},
    Title = {{Emerging social awareness: Exploring intrinsic motivation in multiagent learning}},
    Volume = {2},
    Year = {2011},
    Bdsk-Url-1 = {http://dx.doi.org/10.1109/DEVLRN.2011.6037325}}

  • R. Aylett, M. Vala, P. Sequeira, and A. Paiva, “FearNot! – An Emergent Narrative Approach to Virtual Dramas for Anti-bullying Education,” in Proceedings of the 4th International Conference on Virtual Storytelling, 2007, p. 202–205.

    FearNot! is a story-telling application originally created in the EU FP5 project VICTEC and now extended in the FP6 project eCIRCUS [eCIRCUS 07]. It has applied ideas from Forum Theatre [Boal 79] to the domain of education against bullying. In Forum Theatre, sections of an audience take responsibility for a specific character in the unfolding drama, played by an actor who always stays in role. Episodes in which the actors improvise within an overall narrative framework are broken by interaction sections in which the audience sections talk over with `their’ character what they should do in the next dramatic segment. The actor is free to reject advice that seems incompatible with their role, and may also suspend a dramatic episode if it seems necessary to get further advice.

    @inproceedings{aylett2007icvs,
    Abstract = {FearNot! is a story-telling application originally created in the EU FP5 project VICTEC and now extended in the FP6 project eCIRCUS [eCIRCUS 07]. It has applied ideas from Forum Theatre [Boal 79] to the domain of education against bullying. In Forum Theatre, sections of an audience take responsibility for a specific character in the unfolding drama, played by an actor who always stays in role. Episodes in which the actors improvise within an overall narrative framework are broken by interaction sections in which the audience sections talk over with `their' character what they should do in the next dramatic segment. The actor is free to reject advice that seems incompatible with their role, and may also suspend a dramatic episode if it seems necessary to get further advice.},
    Address = {Berlin, Heidelberg},
    Author = {Aylett, Ruth and Vala, Marco and Sequeira, Pedro and Paiva, Ana},
    Booktitle = {Proceedings of the 4th International Conference on Virtual Storytelling},
    Date-Modified = {2017-05-09 22:19:45 +0000},
    Doi = {10.1007/978-3-540-77039-8_19},
    Editor = {Cavazza, Marc and Donikian, St{\'e}phane},
    Isbn = {978-3-540-77039-8},
    Pages = {202--205},
    Publisher = {Springer Berlin Heidelberg},
    Series = {ICVS 2007},
    Title = {FearNot! -- An Emergent Narrative Approach to Virtual Dramas for Anti-bullying Education},
    Year = {2007},
    Bdsk-Url-1 = {http://dx.doi.org/10.1007/978-3-540-77039-8_19}}

  • [PDF] P. Sequeira and S. Marsella, “Analyzing Human Negotiation using Automated Cognitive Behavior Analysis: The Effect of Personality,” in Proceedings of the 40th Annual Conference of the Cognitive Science Society, 2018, p. 1057–1062.

    @InProceedings{sequeira2018cogsci,
    author = {Pedro Sequeira and Stacy Marsella},
    title = {{Analyzing Human Negotiation using Automated Cognitive Behavior Analysis: The Effect of Personality}},
    booktitle = {Proceedings of the 40th Annual Conference of the Cognitive Science Society},
    year = {2018},
    series = {CogSci 2018},
    pages = {1057--1062},
    address = {Madison, WI, USA},
    date-added = {2018-04-19 23:39:34 +0000},
    date-modified = {2018-04-20 18:47:48 +0000},
    }

Dissertations

  • [PDF] P. Sequeira, “Socio-Emotional Reward Design for Intrinsically Motivated Learning Agents,” Ph.D. Thesis PhD Thesis, Lisbon, Portugal, 2013.

    Reinforcement learning (RL) is a computational approach which models autonomous agents facing a sequential decision problem in a dynamic environment. The behavior of the agent is guided by a reward mechanism embedded into the agent by its designer. Designing flexible reward mechanisms, capable of guiding the agent in learning the task intended by its designer, is a very demanding endeavor: on one hand, artificial agents have inherent limitations that often impact the ability to actually solve the task they were initially designed to accomplish; On the other hand, traditional approaches to RL are too restrictive given the agents limitations, potentially leading to poor performances. Therefore, applying RL in complex problems often requires a great amount of manual fine-tuning on the agents so that they perform well in a given scenario, and even more when we want them to operate in a variety of different situations, often involving complex interactions with other agents. In this thesis we adopt a recent framework for intrinsically-motivated reinforcement learning (IMRL) that proposes the use of richer reward signals related to aspects of the agent’s relationship with its environment that may not be directly related with the task intended by its designer. We propose to take inspiration from information processing mechanisms present in natural organisms to build more flexible and robust reward mechanisms for autonomous RL agents. Specifically, we focus on the role of emotions as an evolutionary adaptive mechanism and also on the way individuals interact and cooperate with each other as a social group. In a series of experiments, we show that the adaptation of emotion-based signals for the design of rewards within IRML allows us to achieve general-purpose solutions and at the same time alleviate some of the agent’s inherent limitations. We also show that social groups of IMRL agents, endowed with a reward mechanism inspired by the way humans and other animals exchange signals between each other, end up maximizing their collective fitness by promoting socially-aware behaviors. Furthermore, by emerging reward signals having dynamic and structural properties that relate to emotions and the way they evolved in nature, we show that emotion-based design might have a greater impact for the adaptation of artificial agents than thought before. Overall, our results support the claim that, by providing the agents with reward mechanisms inspired by the way that emotions and social mechanisms evaluate and structure natural organisms’ interactions with their environment, we provide agent designers with a flexible and robust reward design principle that is able to overcome common limitations inherent to RL agents.

    @phdthesis{sequeira2013phdthesis,
    Abstract = {Reinforcement learning (RL) is a computational approach which models autonomous agents facing a sequential decision problem in a dynamic environment. The behavior of the agent is guided by a reward mechanism embedded into the agent by its designer. Designing flexible reward mechanisms, capable of guiding the agent in learning the task intended by its designer, is a very demanding endeavor: on one hand, artificial agents have inherent limitations that often impact the ability to actually solve the task they were initially designed to accomplish; On the other hand, traditional approaches to RL are too restrictive given the agents limitations, potentially leading to poor performances. Therefore, applying RL in complex problems often requires a great amount of manual fine-tuning on the agents so that they perform well in a given scenario, and even more when we want them to operate in a variety of different situations, often involving complex interactions with other agents.
    In this thesis we adopt a recent framework for intrinsically-motivated reinforcement learning (IMRL) that proposes the use of richer reward signals related to aspects of the agent's relationship with its environment that may not be directly related with the task intended by its designer. We propose to take inspiration from information processing mechanisms present in natural organisms to build more flexible and robust reward mechanisms for autonomous RL agents. Specifically, we focus on the role of emotions as an evolutionary adaptive mechanism and also on the way individuals interact and cooperate with each other as a social group.
    In a series of experiments, we show that the adaptation of emotion-based signals for the design of rewards within IRML allows us to achieve general-purpose solutions and at the same time alleviate some of the agent's inherent limitations. We also show that social groups of IMRL agents, endowed with a reward mechanism inspired by the way humans and other animals exchange signals between each other, end up maximizing their collective fitness by promoting socially-aware behaviors. Furthermore, by emerging reward signals having dynamic and structural properties that relate to emotions and the way they evolved in nature, we show that emotion-based design might have a greater impact for the adaptation of artificial agents than thought before.
    Overall, our results support the claim that, by providing the agents with reward mechanisms inspired by the way that emotions and social mechanisms evaluate and structure natural organisms' interactions with their environment, we provide agent designers with a flexible and robust reward design principle that is able to overcome common limitations inherent to RL agents.
    },
    Address = {Lisbon, Portugal},
    Author = {Sequeira, Pedro},
    Date-Modified = {2017-05-09 22:33:54 +0000},
    Keywords = {Reinforcement Learning, Intrinsic Motivation, Reward Design, Autonomous Agents, Multiagent Systems, Emotions, Cooperation, Evolution, Appraisal, Social Theories},
    Month = {Sep},
    Pages = {196},
    School = {Instituto Superior T\'{e}cnico, Universidade de Lisboa},
    Title = {{Socio-Emotional Reward Design for Intrinsically Motivated Learning Agents}},
    Type = {Ph.D. Thesis},
    Year = {2013}}

Technical Reports

  • [PDF] P. Sequeira, F. S. Melo, and A. Paiva, “Associative Metric for Learning in Factored MDPs based on Classical Conditioning,” GAIPS / INESC-ID Lisbon, GAIPS-TR-002-12, 2012.

    Classical conditioning is a behaviorist paradigm that allows or- ganisms to acquire predictive associations between stimuli in the environment whenever co-occurrences between them are frequent. In this paper we propose a novel associative metric based on the classical conditioning paradigm that, much like what happens in nature, identifies associations between stimuli perceived by a learning agent while interacting with the environment. We use an associative tree structure to identify associations between the perceived stimuli and measure the degree of similarity between states in reinforcement learning (RL) scenarios. Our approach provides a state-space metric that requires no prior knowledge on the structure of the underlying decision problem and which is learned online, i.e., while the agent is learning the RL task it- self. We combine our metric with Q-learning, generalizing the experience of the agent and improving the overall learning per- formance. We illustrate the application of our method in several problems of varying complexity and show that our metric leads to a performance comparable to that obtained with other well- studied metrics but which require full knowledge of the decision problem. The paper concludes by analyzing the impact of our metric in typified conditioning experiments, showing that com- bining our associative metric with standard TD(0) learning leads to the replication of common phenomena described in the classical conditioning literature.

    @techreport{sequeira2012techrep2,
    Abstract = {Classical conditioning is a behaviorist paradigm that allows or- ganisms to acquire predictive associations between stimuli in the environment whenever co-occurrences between them are frequent. In this paper we propose a novel associative metric based on the classical conditioning paradigm that, much like what happens in nature, identifies associations between stimuli perceived by a learning agent while interacting with the environment. We use an associative tree structure to identify associations between the perceived stimuli and measure the degree of similarity between states in reinforcement learning (RL) scenarios. Our approach provides a state-space metric that requires no prior knowledge on the structure of the underlying decision problem and which is learned online, i.e., while the agent is learning the RL task it- self. We combine our metric with Q-learning, generalizing the experience of the agent and improving the overall learning per- formance. We illustrate the application of our method in several problems of varying complexity and show that our metric leads to a performance comparable to that obtained with other well- studied metrics but which require full knowledge of the decision problem. The paper concludes by analyzing the impact of our metric in typified conditioning experiments, showing that com- bining our associative metric with standard TD(0) learning leads to the replication of common phenomena described in the classical conditioning literature.},
    Author = {Sequeira, Pedro and Melo, Francisco S. and Paiva, Ana},
    Date-Added = {2017-05-09 21:58:02 +0000},
    Date-Modified = {2017-05-09 22:33:13 +0000},
    Institution = {GAIPS / INESC-ID Lisbon},
    Month = {June},
    Number = {GAIPS-TR-002-12},
    Title = {Associative Metric for Learning in Factored MDPs based on Classical Conditioning},
    Year = {2012}}

  • [PDF] P. Sequeira, F. S. Melo, and A. Paiva, “Learning by Appraising – An emotion-based approach for intrinsic reward design,” GAIPS / INESC-ID Lisbon, GAIPS-TR-001-12, 2012.

    Reinforcement learning agents have inherent limitations and pose some design challenges that, under certain circumstances, may have an impact on their autonomy and flexibility. A recent frame- work for intrinsically-motivated reinforcement learning proposes the existence of intrinsic reward functions that, if used by the agent during learning, have the potential to improve its perfor- mance when evaluated according to its designer’s goals. Such functions map features of the agent’s history of interaction with its environment into scalar reward values. In this paper, we pro- pose a set of reward features based on four common dimensions of emotional appraisal that, similarly to what occurs in biological agents, evaluate the significance of several aspects of the agent’s history of interaction with its environment. Our experiments in several foraging scenarios show that, by optimizing the relative contributions of each feature for a set of environments of interest, emotion-based reward functions enable better performances when compared to more standard goal-oriented reward functions, par- ticularly in the presence of agent limitations. The results support our claim that reward functions inspired on biological evolution- ary adaptive mechanisms (as emotions are) have the potential to provide more autonomy to learning agents and great flexibility in reward design, while alleviating some limitations inherent to artificial agents.

    @techreport{sequeira2012techrep1,
    Abstract = {Reinforcement learning agents have inherent limitations and pose some design challenges that, under certain circumstances, may have an impact on their autonomy and flexibility. A recent frame- work for intrinsically-motivated reinforcement learning proposes the existence of intrinsic reward functions that, if used by the agent during learning, have the potential to improve its perfor- mance when evaluated according to its designer's goals. Such functions map features of the agent's history of interaction with its environment into scalar reward values. In this paper, we pro- pose a set of reward features based on four common dimensions of emotional appraisal that, similarly to what occurs in biological agents, evaluate the significance of several aspects of the agent's history of interaction with its environment. Our experiments in several foraging scenarios show that, by optimizing the relative contributions of each feature for a set of environments of interest, emotion-based reward functions enable better performances when compared to more standard goal-oriented reward functions, par- ticularly in the presence of agent limitations. The results support our claim that reward functions inspired on biological evolution- ary adaptive mechanisms (as emotions are) have the potential to provide more autonomy to learning agents and great flexibility in reward design, while alleviating some limitations inherent to artificial agents.},
    Author = {Sequeira, Pedro and Melo, Francisco S. and Paiva, Ana},
    Date-Added = {2017-05-09 21:59:37 +0000},
    Date-Modified = {2017-05-09 22:33:01 +0000},
    Institution = {GAIPS / INESC-ID Lisbon},
    Month = {March},
    Number = {GAIPS-TR-001-12},
    Title = {Learning by Appraising - An emotion-based approach for intrinsic reward design},
    Year = {2012}}