Application of hemodynamic prefrontal cortex desirability signals as reinforcers in machine learning

Abstract

Decision-making ability in the frontal lobe (among other brain structures) relies on the assignment of value to states of the organism and its environment. Then higher valued states can be pursued and lower valued (or negative) states avoided. The same principle forms the basis for computational reinforcement learning controllers, which have been fruitfully applied both as models of value estimation in the brain, and as artificial controllers in their own right. This work shows how state value signals decoded from frontal lobe hemodynamics, as measured with near-infrared spectroscopy (NIRS), can be applied as reinforcers to an adaptable artificial learning agent in order to guide its acquisition of skills. A set of experiments carried out on an alert macaque demonstrate that both oxy- and deoxyhemoglobin concentrations in the frontal lobe show differences in response to both primarily and secondarily desirable (versus undesirable) stimuli. This difference allows a NIRS signal classifier to serve successfully as a reinforcer for an adaptive controller performing a virtual tool-retrieval task. The agent’s adaptability allows its performance to exceed the limits of the NIRS classifier decoding accuracy. We also show that decoding state desirabilities is more accurate when using relative concentrations of both oxyhemoglobin and deoxyhemoglobin, rather than either species alone. This suggests that studies of value-related brain function that infer patterns of neural activity based solely on deoxyhemoglobin concentration (as in fMRI) may yet be incomplete. This work is part of a continuing investigation into the use of reinforcement learning agents as controllers in brain-machine interfaces.