Effect of lysergic acid diethylamide (LSD) on reinforcement learning in humans

This placebo-controlled study (n=19), which also used computational modeling, argues that LSD (75μg) increased reward learning rates heightened plasticity, which in turn could be the mechanism through which psychedelics help reshape maladaptive (‘stuck’) patterns.

Abstract

“Background The non-selective serotonin 2A (5-HT_2A) receptor agonist lysergic acid diethylamide (LSD) holds promise as a treatment for some psychiatric disorders. Psychedelic drugs such as LSD have been suggested to have therapeutic actions through their effects on learning. The behavioural effects of LSD in humans, however, remain incompletely understood. Here we examined how LSD affects probabilistic reversal learning (PRL) in healthy humans.

Methods Healthy volunteers received intravenous LSD (75 μg in 10 mL saline) or placebo (10 mL saline) in a within-subjects design and completed a PRL task. Participants had to learn through trial and error which of three stimuli was rewarded most of the time, and these contingencies switched in a reversal phase. Computational models of reinforcement learning (RL) were fitted to the behavioural data to assess how LSD affected the updating (‘learning rates’) and deployment of value representations (‘reinforcement sensitivity’) during choice, as well as ‘stimulus stickiness’ (choice repetition irrespective of reinforcement history).

Results Raw data measures assessing sensitivity to immediate feedback (‘win-stay’ and ‘lose-shift’ probabilities) were unaffected, whereas LSD increased the impact of the strength of initial learning on perseveration. Computational modelling revealed that the most pronounced effect of LSD was the enhancement of the reward learning rate. The punishment learning rate was also elevated. Stimulus stickiness was decreased by LSD, reflecting heightened exploration. Reinforcement sensitivity differed by phase.

Conclusions Increased RL rates suggest LSD induced a state of heightened plasticity. These results indicate a potential mechanism through which revision of maladaptive associations could occur in the clinical application of LSD.”

Authors: Jonathan W. Kanen, Qiang Luo, Mojtaba R. Kandroodi, Rudolf N. Cardinal, Trevor W. Robbins, David J. Nutt, Robin L. Carhart-Harris & Hanneke E. M. den Ouden

Summary

Researchers from the University of Cambridge, Behavioural and Clinical Neuroscience Institute, Fudan University, Shanghai 200433, PR China, the University of Tehran, School of Electrical and Computer Engineering, Radboud University, Nijmegen, The Netherlands, and the Imperial College London conducted research on psychedelics.

Abstract

LSD, a non-selective serotonin 2A (5-HT2A) receptor agonist, increases the reward learning rate and decreases the punishment learning rate in healthy humans. This suggests that LSD induces a state of heightened plasticity through which revision of maladaptive associations could occur.

Introduction

LSD has been shown to improve learning and plasticity in non-human animals, but few studies have examined its effect on human behaviour. Here we tested whether LSD altered probabilistic reversal learning in healthy volunteers, and explored how LSD altered underlying learning mechanisms, using reinforcement learning models.

Serotonin is involved in adapting behaviour flexibly as environmental circumstances change, as well as processing aversive outcomes. Profound neurotoxin-induced depletion of serotonin from the marmoset orbitofrontal cortex causes perseverative, stimulus-bound behaviour, and acute administration of selective serotonin reuptake inhibitors (SSRIs) causes increased sensitivity to negative feedback.

LSD affects the serotonin system and has dopamine type 2 receptor agonist properties. Dopamine is also involved in learning from feedback and is also involved in instrumental reversal learning.

The current study examined the effects of LSD on learning in humans. It found that LSD altered the rate at which value was updated following reward or punishment, and also altered the degree to which behaviour was stimulus-driven.

Results

We verified that LSD did not impair participants’ overall ability to perform the task and examined whether LSD affected the number of times each stimulus was chosen. We found no interaction of LSD with stimulus or phase.

LSD enhanced the relationship between initial learning and perseveration, and making fewer errors during the acquisition phase predicted more perseverative errors when on LSD but not when under placebo.

We next assessed whether LSD influenced individuals’ responses on trials immediately after positive versus negative feedback. We found no main effect of LSD or interaction of valence and LSD.

We fitted and compared three reinforcement learning models, and found that a model with four parameters best described behaviour. The four parameters were reward learning rate, punishment learning rate, reinforcement sensitivity, and stimulus stickiness.

LSD increased the reward learning rate and the punishment learning rate compared to placebo, but not the tendency to ‘stay’ following reward or punishment.

Discussion

In the current study, we found that LSD increased learning rates and the impact of previously learnt values on subsequent perseverative behaviour.

Psychedelic drugs have been hypothesised to destabilise pre-existing beliefs, which is directly compatible with increased reinforcement learning rates. This finding is particularly important for understanding the mechanisms through which LSD might be therapeutically useful.

LSD has a broad effect on a range of neurotransmitter systems, including the serotonin and dopamine systems, and is believed to promote psychological plasticity through action at 5-HT2A receptors and downstream enhancement of NMDA receptor transmission and brain-derived neurotrophic factor expression.

LSD affects the serotonin system and dopamine receptors, and this may explain why dopaminergic manipulations alter learning rates.

Serotonin – dopamine interactions could underlie the present findings, as stimulation of 5-HT2A receptors in the prefrontal cortex increased ventral tegmental area dopaminergic activity. However, a previous study in rodents showed that LSD had consistent effects across four different time lags.

LSD impaired flexibility, and this was blocked by the 5-HT2A antagonist ketanserin. This suggests that LSD “stamps in” new learning following drug administration, which may subsequently be harder to update.

LSD’s effects to induce cognitive inflexibility are at odds with the observation that LSD enhanced plasticity (through enhanced learning rates). However, when initial learning and tests of cognitive flexibility were conducted before LSD administration, LSD improved reversal learning.

LSD enhanced the rate at which humans updated their beliefs based on feedback. This finding has implications for understanding how LSD might be therapeutically useful.

Methods

Nineteen healthy volunteers over the age of 21 received either intravenous LSD (75 g in 10 mL saline) or placebo in a single-blind within-subjects balanced-order design. They provided written informed consent after briefing on the study and screening. Screening was conducted at the Imperial College London Clinical Research Facility at the Hammersmith Hospital campus, and the PRL task was administered approximately five hours after injection.

In a novel version of a widely used PRL task, participants were required to choose from three visual stimuli, presented at three of four randomised locations on a computer screen. The most and least optimal stimuli reversed after 40 trials, to distinguish learning to select the mostly rewarding stimulus from learning to avoid the mostly punishing stimulus.

LSD impaired participants’ ability to perform a task by reducing their feedback sensitivity (win-stay and lose-stay).

Perseveration was assessed based on responses in the reversal phase, and was measured using the win-stay / lose-shift metric. The first trial in the reversal phase was excluded from the perseveration analysis, as at that point behaviour cannot yet be shaped by the new feedback structure.

We fitted three reinforcement learning models to the behavioural data using a hierarchical Bayesian method, via Hamiltonian Markov chain Monte Carlo sampling implemented in Stan 2.17.2. Convergence was checked according to the potential scale reduction factor measure, and models were compared via a bridge sampling estimate of the marginal likelihood.

The Bayesian hierarchy consisted of drug condition at the highest level, and subject at the lowest level. The intersubject variability was merged with the mean of each drug condition, and the priors were unbiased with respect to LSD versus placebo.

We calculated the change (LSD – placebo) in parameters for each of the 8,000 simulation runs and tested them against zero via the HDI.

We tested the hypothesis that positive versus negative feedback guides behaviour differentially, and that LSD affects this. We augmented a basic RL model with separate learning rates for reward rew and punishment pun, and added a reinforcement sensitivity parameter.

The reinforcement sensitivity parameter ( reinf ) was calculated from the probability values of choosing stimulus 1 for n = 3 choice options.

Model 2 augmented a simple RL model by also describing the tendency to repeat a response, irrespective of the outcome that followed it. This model was used to test the hypothesis that LSD affects this basic perseverative tendency. The final quantity controlling choice incorporated the additional stickiness parameter as Qt = Qreinft + Qstimt, and was fed into the softmax function as above.

Model 3 tested the hypothesis that LSD affects how positive versus negative feedback guides behaviour differentially, and how LSD affects a basic perseverative tendency.

Figures

Figure 1 shows the probabilistic reversal learning task, where subjects chose one of three stimuli and were given positive feedback with a 75% probability, negative feedback with 50%, and no feedback with 25%. The mean per-subject probability of choosing each stimulus during the acquisition and reversal phases are shown, collapsed across LSD and placebo sessions. The probability of repeating a choice after receiving positive or negative feedback is shown.

Figure 2 shows that subjects who took LSD exhibited more perseverance than those who took placebo, as indicated by the higher mean probability of choosing each stimulus and the higher probability of staying after receiving positive or negative feedback.

Tables

T.W.R., R.N.C., H.E.M.d.O., and J.W.K. declare no conflicts of interest. R.L.C-H, Q.L., and M.R.K. have received research grants from Shionogi & Co and GlaxoSmithKline.

This study was funded by the Walacea.com crowdfunding campaign, the Beckley Foundation, the Gates Cambridge Scholarship, the Wellcome Trust, the Netherlands Organisation for Scientific Research, H.E.M.d.O., and the UK Medical Research Council.

Supplementary information

We simulated 100 subjects using the posterior mean parameters from the winning model to determine how behavioural patterns in the synthetic data compared to the raw data.

Simulations were run using parameter estimates from the winning model to assess whether the model could capture the observed effects of LSD on raw behaviour. The results showed that LSD did not affect lose-stay probability, acquisition performance, or perseveration.

Supplementary Figure 1 shows simulated data for initial learning and perseveration on LSD versus placebo, as well as the probability of choosing different stimuli while under placebo and LSD, and the probability of repeating a choice after receiving positive or negative feedback.