Great Expectations: Recommendations for improving the methodological rigor of psychedelic clinical trials

This article (2021) explores the methodological challenges faced in psychedelic clinical trials and recommendations for future research. Several sources of potential bias exist; difficulties blinding, inflated expectations as well as placebo and nocebo effects, amongst others. Recommendations to avoid these biases include; study development, participant recruitment and selection, incomplete disclosure of the study design, choice of active placebo condition, as well as the measurement of participant expectations and masking efficacy.

Abstract

“Rationale: Psychedelic research continues to garner significant public and scientific interest with a growing number of clinical studies examining a wide range of conditions and disorders. However, expectancy effects and effective condition masking have been raised as critical limitations to the interpretability of the research.

Objective: In this article, we review the many methodological challenges of conducting psychedelic clinical trials and provide recommendations for improving the rigor of future research.

Results: We found that although some challenges are shared with psychotherapy and pharmacology trials more broadly, psychedelic clinical trials have to contend with several unique sources of potential bias. The subjective effects of a high-dose psychedelic are often so pronounced that it is difficult to mask participants to their treatment condition; the significant hype from positive media coverage on the clinical potential of psychedelics influences participants’ expectations for treatment benefit; and participant unmasking and treatment expectations can interact in such a way that makes psychedelic therapy highly susceptible to large placebo and nocebo effects. Specific recommendations to increase the success of masking procedures and reduce the influence of participant expectancies concern study development, participant recruitment and selection, incomplete disclosure of the study design, choice of active placebo condition, as well as the measurement of participant expectations and masking efficacy.

Conclusion: Incorporating these design elements is intended to reduce the risk of bias in psychedelic clinical trials and thereby increase the ability to discern treatment-specific effects of psychedelic therapy.”

Authors: Jacob Aday, Boris D. Heifets, Steven D. Pratscher, Ellen Bradley, Raymond Rosen & Joshua D. Woolley

Notes

The therapeutic potential of psychedelics is garnering significant attention from academics, elected officials and the public alike. Thanks to solid clinical research, the narrative surrounding these psychoactive substances is changing for the better. The medicalization of psychedelics like psilocybin, MDMA and ketamine is slowly becoming a reality. If therapeutic models using these substances are truly going mainstream, they must withstand the scrutiny of the clinical trial process and address the inherent methodological issues which often occur conducting such trials.

For example, the proliferation of news articles regarding the promise of psychedelic-assisted therapies is paradoxically leading to expectancy effects among clinical trial participants i.e overwhelmingly positive news articles are leading people to believe that psychedelic-assisted therapy will ‘cure’ any mental health problem they may be experiencing. In reality, psychedelics will not work for everyone and are by no means a silver bullet aimed at global mental health.

A recent preprint article explores some of the major methodological issues faced when conducting clinical trials with psychedelics. As well as the issues surrounding expectancy effects, the researchers identified several unique sources of potential bias when conducting such trials:

• The subjective effects of a high-dose psychedelic are often so pronounced that it is difficult to mask participants to their treatment condition.
• The significant hype from positive media coverage on the clinical potential of psychedelics influences participants’ expectations for treatment benefit.
• Participant unmasking and treatment expectations can interact in such a way that makes psychedelic therapy highly susceptible to large placebo and nocebo effects.

The researchers in the present study recommend a number of design elements for future research in order to minimize the risk of bias:

• Study development and design; a 3-arm study is best to test for substance efficacy (psychedelic vs. active placebo vs. inactive placebo) & a single administration session of a psychedelic is recommended.
• Participant recruitment and selection; recruit psychedelic-naïve participants in order to prioritize effective condition masking.
• Expectation bias; investigators should emphasize the uncertainty regarding treatment efficacy & use established measures of expectancy e.g the Stanford Expectations of Treatment Scale.
• It is critically important to distinguish “incomplete exposure” from “deception.”
• Anticipating the placebo effect, measuring the contribution of expectancies, assessing the effectiveness of masking, and systematically reporting these data will set standards and lead to iterative improvements in trial design.

The present study seeks to improve the clinical trial process involving psychedelics. Legitimatizing psychedelic research is no mean feat. However, addressing the issues identified in this study and taking the authors recommendations into account will help to bring psychedelic-assisted therapies to the masses.

Summary

Psychedelic clinical trials continue to garner significant public and scientific interest, but expectancy effects and effective condition masking have been raised as critical limitations.

Overview

Recent high-profile clinical trials with psychedelic drugs have highlighted methodological challenges related to rigorous study design and condition masking that have simmered in both psychotherapy and pharmacology research for decades. This paper provides practical methodological recommendations for improving future psychedelic and non-psychedelic research.

Treatment‑nonspecific effects

In a clinical study, changes in symptoms can be observed because of treatment-specific or treatment-nonspecific effects. Control conditions allow the trialist to filter out contributions of treatment-nonspecific effects from treatment-specific effects.

The natural history of any given disease under study may be the least controllable source of treatment-nonspecific change that can confound clinical trial interpretation. Regression to the mean is a ubiquitous statistical phenomenon that results whenever cases are selected for follow-up based on abnormally high or low scores at baseline.

Participants can change their behavior simply as a consequence of the interest, care, or attention received as part of a study.

Clinical trials are under increased scrutiny and observation as compared to an unobserved clinical setting, and this may impact both the quality and quantity of patient care. Additionally, repeated observation and measurement of behaviors and symptoms can alter those same behaviors and symptoms.

There are many mechanisms by which symptoms may change in a clinical trial irrespective of the treatment being tested. Therefore, it is important to include control arms that do not receive the treatment. Participants often have expectations regarding the efficacy of the treatment under study. These expectations can affect their clinical outcomes, and baseline expectancies about a treatment’s therapeutic effects can also impact masking efficacy.

Expectancies in psychotherapy and pharmacology research

Tambling (2012) differentiates between expectations about the process of treatment and expectations about the outcome of treatment. In psychedelic drug trials, participants may have high expectations about the acute effects of the drugs.

Studies suggest that outcome expectancies are stronger predictors of therapeutic effects than specific psychotherapy techniques, and that positive outcome expectancies are related to stronger alliance with the therapist, which is associated with better treatment results.

Negative outcome expectations can also influence clinical outcomes. For example, participants who believe they have been assigned to a treatment that they believe is unlikely to improve their symptoms are more likely to have worse outcomes than those who believe they have been assigned to an active treatment.

Waitlist control designs may artificially inflate intervention effect size estimates, as demonstrated by the higher anxiety scores of waitlisted participants after the treatment period compared to the beginning.

Psychotherapy and pharmacology studies rarely measure participant outcome expectancies, and the few studies that do measure expectancies use brief and study-specific measures. There is no standard for addressing participant expectations.

Psychedelic research and expectations

Modern psychedelic therapy clinical trials involve an arduous screening process, multiple preparation sessions, single or multiple drug dosing sessions, and integration sessions after drug administration. The participants work with the same clinical team in integration sessions to make meaning of their experiences. Psychedelic therapy includes aspects of both pharmacology and psychotherapy, and a participant’s process expectations and outcome expectations are subject to change as they gather more information about possible drug effects and experience the actual drug effects.

Participants’ expectations as well as intentions are thought to play a prominent role in the psychedelic drugs’ acute and long-term effects, and may reconcile the paradoxical conceptions that have been held about the drugs. Psychedelic therapy may be effective, but it can also be co-opted for nefarious purposes. Therefore, it is important to include an enhanced informed consent process about possible belief changes induced by psychedelic therapy prior to enrolling participants into a clinical trial.

Although many studies have found that pre-dosing expectations are integral to the effects of psychedelics, very few have actually measured them. A single study found that participants with positive expectancies experienced greater decreases in neuroticism and greater increases in extraversion and conscientiousness following treatment. A recent systematic review found that those with a recreational intention with psychedelics tended to have less challenging experiences when they used a psychedelic, suggesting that expectations may influence the drug’s effects.

High-dose psychedelic trials may be particularly susceptible to a type of bias termed “hype” or the “Michael Pollan effect”, which is a combination of heightened positive expectations and functional unmasking. This makes identification of a treatment-specific effect in high-dose psychedelic trials particularly challenging.

Certain characteristics of psychedelic drug trials, such as the use of two therapists at a time and rituals like placing a fresh rose in the room on dosing day, may contribute to enhanced expectations.

Modern era clinical research design elements

Open-label study designs are the most resemble real-world, non-research settings, but do not control for most confounding nonspecific factors that can affect clinical outcomes.

Some treatment-nonspecific factors can be controlled by selecting cases based on multiple baseline observations, using ANCOVA modeling, or applying a correction formula. A waitlist control condition may also be considered.

Double-blind randomized controlled trials (RCTs) are the gold standard design for identifying a true treatment-specific effect, and are best achieved with active placebo comparators, in which the control condition closely resembles the presentation and side effects of the experimental treatment without providing the therapeutic effects.

There has been considerable debate about what constitutes a proper “inert” placebo for psychotherapy, and improved masking procedures must be implemented into psychedelic science for the field to meet the assumptions of the current gold standard clinical trial design.

Crossover RCT designs are used in many pharmacological studies as an efficient way to account for treatment-nonspecific confounds, but they have the potential for carryover effects and masking issues. Thus, a parallel (between-subjects) RCT design may be more suitable for psychedelic trials.

We have repeatedly noted the importance of adequate masking in double-blind RCTs, but many researchers report their studies as being “double-blind” without testing such claims. A systematic review of methods of masking in randomized controlled trials with pharmacologic treatments concluded that reporting of condition masking is generally “quite poor,” and a high proportion of studies are effectively unmasked. Similarly, a comprehensive literature search found that masking was not maintained in 20/23 “double-blind” studies.

Masking attempts in psychedelic studies

Multiple approaches have been attempted to address methodological challenges specific to psychedelic trials, including active placebos, which have generally failed to mask participants and therapists to treatment conditions. Despite the clear masking failure of niacin, researchers today still use it as the active placebo in clinical trials with psychedelics, although the success of masking is typically less than 25% or unreported. Researchers used a mixture of coco powder, vitamins, turmeric powder, quinoa, traces of coffee, and potato flour as a placebo to mimic the effects of ayahuasca, but most participants were still able to accurately identify their treatment assignment.

Low doses of psychedelics have been tried as a potential control condition to improve participant masking. A low dose of psilocybin as part of an active control condition may be a promising starting point, but it may also be limited by ethical considerations and the possibility that low-dose psychedelics will agitate people without allowing them to “breakthrough”.

Some studies attempt to mask the study design by not disclosing the number of treatment arms to participants, or by having participants consent to receiving one or more substances, but this approach has not proven to be particularly effective.

Muthukumaraswamy et al. (2021) suggested that active placebos may need to be combined with alternative trial designs, and that participants need to be given vague information about the acute effects of psychedelics when consenting. Another recommendation provided was the 2 x 2 balanced placebo design, which offers a potentially rigorous experimental means for separating pharmacological effects of the drug from participant expectations. Researchers have begun to address the methodological challenges associated with masking, treatment expectations, and their combined impact on study results.

Mental confounds related to expectancies and placebo effects in psychedelic studies largely stem from inadequate masking. Our recommendations focus on improving masking in psychedelic trials through a combination of procedures intended to decrease participants’ confidence in their assigned treatment arm.

Study development and design

A study design should include a control condition, a number of study arms, and a specific purpose. An open-label study design may be appropriate for examining safety, feasibility, or proof-of-concept, but not for examining treatment efficacy.

A 3-arm design (which compares the treatment and active placebo conditions with the inactive control condition) is a promising way to disentangle placebo effects, but a recent systematic review challenges the notion that this design actually reduces the measured placebo response.

We recommend between-subjects designs with a single psychedelic administration when evaluating treatment efficacy, because multiple psychedelic dosing sessions may have therapeutic advantages, but the current controversies surrounding psychedelic therapy are focused on whether there is any drug-specific benefit of the complex therapeutic intervention.

Several trials have included an open-label crossover component, but we disagree with the idea that this design feature is ethically mandatory. We recommend incorporating well-established strategies to minimize harm to participants, such as stopping rules and adaptive randomization based on outcome.

Participant recruitment and selection

We recommend recruiting psychedelic-naive participants for clinical trials, because they will have less confidence in the active placebo, and will have less experience with the treatment condition. Experienced hallucinogen users are highly accurate at differentiating between whether they received psilocybin or DXM, but those without prior hallucinogen use may be easier to convince. However, the phenomenological experience and intensity of drug effects may differ in first-time users, which may limit generalizability.

Outcomes, assessments, and endpoints

The choice of outcome measures, assessments, and endpoints for psychedelic clinical trials can have a large impact on the evaluation of treatment benefit and overall methodological rigor.

Psychedelic therapy, regardless of any placebo group difference, are likely more important to patients, providers, and stakeholders than an acute improvement that is not maintained. However, rigorous, well-controlled trials are needed to define clear evidence of benefit.

We recommend using multiple methods of measurement to comprehensively examine the effects of psychedelic therapy in clinical trials. These measures include patient-reported outcomes (PROs), clinician-administered assessments or observer reports, and biomarkers and/or behavioral tasks that reflect component processes related to the index pathology. Two categories of biomarkers are recognized by the FDA: predictive biomarkers and surrogate endpoints. Predictive biomarkers indicate whether certain participants respond differentially to the treatment or placebo, and surrogate endpoints include accurate and well-validated lab measures or physical signs that reliably predict clinically meaningful endpoints.

Study procedures: managing and measuring treatment expectations

At the beginning stages of a study, investigators should emphasize the uncertainty regarding the treatment efficacy and explain neutrally that there is significant variability between people. Moreover, all study staff should be masked to treatment arm assignment and trained to present the study neutrally.

To measure participants’ treatment expectations, use the Stanford Expectations of Treatment Scale or the Credibility and Expectancy Questionnaire. Face-valid questions can also be used, as well as semi-structured interviews.

Participants’ positive and negative treatment expectations should be assessed during the preparation and integration sessions, and after the drug dosing session. This may be useful in predicting clinical outcomes.

Study procedures: incomplete disclosure

We reviewed studies where incomplete disclosure was used to reduce participants’ certainty regarding their treatment assignment. In designing a trial, it is critical to distinguish between “incomplete disclosure” and “deception”, as the former requires thorough scientific justification and authorization by institutional review boards. If it is considered ethically appropriate, withholding information from participants and study staff about the number of study arms and the exact doses administered may be particularly effective for enhancing masking success.

Researchers may want to list all potential side effects of the study drugs together, to reduce participants’ confidence in their treatment arm assignment, while still fully informing them of all the potential drug effects they may be exposed to.

Incomplete disclosure to participants and study personnel regarding key elements of a study’s design may help to minimize the impact of beliefs on study outcomes. However, maintaining uncertainty about the true study design is challenging and requires appropriate debriefing protocols and masking assessments.

Study procedures: active placebo

Researchers have tried to use a drug that mimics psychedelic effects but provides no therapeutic benefit in psychopharmacology studies. However, the internal contradiction in this strategy becomes apparent if the subjective effects produced by psychedelics themselves drive therapeutic benefit.

A recent study found that an opioid antagonist blocked ketamine’s antidepressant and anti-suicidal effects, but had minimal effect on ratings of ketamine-induced dissociation. This finding suggests that pharmacological agents may be discovered that interrupt neuroplastic processes triggered by psilocybin, but do not interfere with its acute psychedelic effects. Another highly innovative approach to psychedelic research pairs psilocybin with midazolam, a benzodiazepine, and focuses on safety. The broader hypothesis is that participants who do not form memories of the experience do not experience therapeutic benefit.

Researchers note several challenges in using amnestic agents in psychiatric populations, including uncertainty as to whether midazolam retains its amnestic property when paired with a psychedelic.

Psychedelic therapy may be an uninterruptible whole, and active placebos are needed to adequately mask psychedelic effects. Ketamine, DXM, and high doses of tetrahydrocannabinol may be compelling options, especially when combined with drug-naive participants.

Design of an active placebo should be considered in concert with other study design elements, to reduce expectancy effects across treatment arms. For simplicity, a study could be designed as a two-arm comparison of high-dose psilocybin versus ultra-low-dose (ineffective) psilocybin plus an active placebo. Participants would be truthfully informed of all the possible treatment conditions and may be reasonably uncertain of their treatment assignment.

Analysis: assessing and reporting outcomes related to trial design

The placebo effect and effective masking are key considerations for designing an interpretable study involving psychoactive drugs. Calculating statistical power based on known placebo effect sizes is recommended.

Estimating the size of the placebo effect in a trial informs statistical power calculations, which may impact the feasible number of treatment arms. A common method of partitioning the placebo effect from the treatment effect is to compare against a “placebo benchmark”. Investigators can minimize some components of the placebo effect by repeating baseline assessment, only enrolling participants with stable response characteristics, and measuring the factors that make up the placebo effect.

To minimize the influence of masking, investigators should create protocols and adherence plans for all relevant study staff, and measure participant- and therapist-perceived treatment allocation, certainty of treatment allocation, and the reason for their guess both immediately after the psychedelic dosing session(s) and at the end of the study.

Modern adaptive trial designs can help investigators at least achieve an even distribution of biases across conditions, but balancing recruitment on treatment outcome expectations must be weighed against other recruitment priorities.

In a study measuring pleasantness of affective touch, researchers found that MDMA was significantly more effective than an active placebo, but adding baseline expectations as a covariate to the analyses reduced the effect of MDMA on affective touch.

Beyond the scope of RCTs

One notion to consider is embracing expectancy and placebo effects, which could be tailored to psychedelic treatments and be particularly synergistic with them. This would mean that the current gold standard clinical trial design may not be sensitive to detecting the therapeutic effect of an individual treatment element.

A potential solution to this dilemma may be to shift focus from efficacy trials to pragmatic clinical trial designs, which have an alternative goal of assessing treatment effectiveness. These trials would include one or more alternative therapies to the treatment under study.

A closely related approach to testing the effectiveness of psychedelic therapy is to evaluate large-scale population data using so-called “natural experiments”. If positive trends in mental health are observed at the population level after the introduction of legal psychedelic therapy, the role of expectations may be considered immaterial.

Conclusion

Psychedelic studies are particularly challenging because of the additional confounds related to “hype” and salient psychoactive effects. Innovative, disruptive experimental designs may be needed to separate pharmacological effects from multiple, interactive socio-psychological influences in psychedelic medicine.

This review highlights that traditional placebo masking with inert comparators is insufficient for high-dose psychedelic studies, and that this issue often extends to psychotherapy and pharmacology research more broadly. Several recommendations are presented for improving the methodological rigor of future psychedelic studies.