On the fallibility of placebo control and how to address it: a case study in psychedelic microdosing

This preprint (2022) uses computational modelling to show that the combination of weak blinding and positive treatment expectancy can lead to activated expectancy bias (AEB), which is an uneven distribution of expectancy effects between the treatment arms due to patients recognizing their treatment allocation in psychedelic microdosing randomized controlled trials. The results demonstrate that a placebo control group is in itself not sufficient to control for expectancy effect and that placebo-controlled studies are more fallible than conventionally assumed.

Abstract

“In medical research, the gold standard experimental design is the blinded randomized controlled trial. Despite the central role of blinding, it is rare for trials to assess blinding integrity and to incorporate this information into the interpretation of results. Here we use computational modelling to show that the combination of weak blinding and positive treatment expectancy can lead to activated expectancy bias (AEB), which is an uneven distribution of expectancy effects between the treatment arms due to patients recognizing their treatment allocation. We show that this bias can inflate estimates of treatment effects and potentially create false positive findings. To counteract this bias, we introduce the Correct Guess Rate Curve (CGRC), a novel analytical tool that can estimate what would be the outcome of a perfectly blinded trial based on data from an imperfectly blinded trial. We apply CGRC to pseudo-experimental data generated by our computational model and show that the method produces AEB corrected results. Furthermore, to demonstrate the impact of AEB and the utility of the CGRC on empirical data, we re-analyzed data from a previously published self-blinding microdose trial. Results suggest that the observed placebo vs. microdose differences are susceptible to AEB, therefore, at risk of being false positives. These results demonstrate that a placebo control group is in itself not sufficient to control for expectancy effects, arguing that placebo-controlled studies are more fallible than conventionally assumed, which has implications for evidence-based medicine and numerous public health policies.”

Authors: Balázs Szigeti, David Nutt, Robin Carhartt-Harris & David Erritzoe

Summary

We use computational modelling to show that weak blinding and positive treatment expectancy can lead to activated expectancy bias, which can inflate estimates of treatment effects and potentially create false positive findings. We introduce the Correct Guess Rate Curve (CGRC), an analytical tool that can counteract this bias. Results suggest that placebo-controlled studies are more fallible than conventionally assumed, and therefore, at risk of being false positives.

In medical research, blinded randomized controlled trials are the generally accepted gold standard experimental design. However, only 2%-7% of trials assess and report blinding integrity, and more concerning, over 50% of trials feature imperfect blinding and are thus more fallible than conventionally assumed.

Poor reporting of blinding integrity may be explained by at least three factors, including the lack of an accepted standard for how to assess blinding integrity, and a reluctance to assess blinding due to a fear that weak blinding could cast doubt on positive trial outcomes.

Recently, ‘microdosing’ has emerged as a new paradigm for psychedelic use, which involves taking low, sub-hallucinogenic, doses of psychedelic substances 1-4 times a week. Observational studies have generally confirmed the positive anecdotal claims, but placebo-controlled studies have failed to find robust evidence for larger than placebo efficacy.

We conducted a self-blinding citizen science trial on microdosing that allowed us to obtain a large sample size while implementing placebo control at minimal logistic and economic costs.

In this manuscript, we first describe our computational model of activated expectancy, then apply the Correct Guess Rate Curve (CGRC) to pseudo-experimental data, and finally apply CGRC to data from the self-blinding microdose study.

Participants have higher efficacy expectations for the active treatment than for the placebo treatment, i.e., treatment expectancy bias. This expectation difference is represented by the arrow between the treatment (TRT) and perceived treatment (PT) nodes on Figure 1.

The model was used to generate pseudo-experimental data with 119 parameters and 50 trials with 120 participants.

A linear model was used to evaluate the treatment effect for both the pseudo-experimental data generated from computational models and the microdose data.

Participants in the self-blinding microdose trial took their own placebos, which were encased inside non-transparent gel capsules. The researchers tracked when the placebo was taken without sharing this information with participants.

The trial’s acute and post-acute outcomes were re-analyzed, including the positive and negative affect schedule, cognitive performance score, visual analogue scale items for mood, energy, creativity, focus, and temper, and the Warwick – Edinburgh mental well-being scale.

Participants made a binary choice guess whether their capsule was placebo or microdose. If this knowledge is considered, a random guesser with a correct guess rate of 0.67 is expected, whereas if it is not considered, a correct guess rate of 0.5 is expected.

We developed a novel analytical technique called Correct Guess Rate Adjustment, which can estimate the outcome of a perfectly blinded trial, based on data from an imperfectly blinded trial. The Correct Guess Rate Curve is a generalization of the Correct Guess Rate Adjustment.

A trial with CGR=0.55 can be approximated by drawing 0.55n random samples from the correct guess KDE and 0.45n random samples from the incorrect guess KDE, etc.

CGRCs can be generated using the conda computational environment, and the data analyzed and scripts to reproduce all figures and major statistical findings described here.

The activated expectancy bias model was used to generate pseudo-experimental data with 2*2=4 configurations of parameters. The data were examined with both traditional (non-CGR adjusted) and CGR-adjusted analysis, and a true positive effect of 214 treatment was robustly identified by both traditional and CGR adjusted models.

The CGR-adjusted models had a significantly lower average treatment p-value than the traditional models, and the true positive treatment effect was robustly identified by both models. The traditional analysis overestimated the effect by 33%. Traditional analysis of a pseudo-experiment that does not adjust for compromised blinding overestimates treatment effects and can even create false positive findings when activated expectancy bias is present. CGR-adjusted analysis avoids these mistakes and recovers the true trial results.

Participants with the same expectancy regarding the active and placebo treatments had the same average treatment p-value.

We advanced from analyzing pseudo-experimental data to scrutinizing empirical data from the self-blinding microdose trial. We found that microdosing increased self-perceived energy beyond what is explainable by expectancy effects, although the magnitude of the effect is small.

Effective blinding distributes expectancy biases equally between treatment arms, but in practice, blinding integrity is rarely effective, meaning that participants can deduce their treatment allocation at a better than chance rate. If participants can deduce their treatment allocation, then treatment expectancy can bias outcomes.

The co-occurrence of weak blinding and treatment expectancy bias is necessary for activated expectancy bias to occur in medical trials. Blinding integrity is assessed in only 2%-7% of trials, and is found to be ineffective in over 50% of the trials. In the self-blinding microdose trial, traditional, non-CGR adjusted analysis yielded statistically significant placebo vs. microdose differences on several scales, but these findings are at risk of being false positive findings created by AEB. In our view, placebo-controlled trial data should only be considered ‘gold standard’ if blinding integrity is demonstrated with empirical data. The Correct Guess Rate Curve (CGRC) is a statistical procedure that can be used to counteract activated expectancy bias, which is driven by weak blinding.

Most solutions to improve blinding are difficult to implement and resource intensive, for example active placebos and suggestive messaging.

The current work demonstrates an alternative solution to overcome weak blinding: the correct guess rate curve (CGRC) can be applied to any trial if patients’ treatment guess was collected. This method is less resource intensive than traditional data analysis. When analyzing self-blinding microdose trials, investigators need to consider the dominant source of unblinding in their trial prior to using CGRC. In our view, the energy VAS remained significant after CGR adjustment, with a 40% reduced effect size. Activated expectancy bias may explain the observed placebo vs microdose differences, since the study had weak blinding and positive treatment expectations, and participants had multiple past psychedelic experiences.

The self-blinding microdose trial results should be interpreted with caution, especially if evidence for effective blinding is not presented. Participants reported agitation, muscle tension, stomach discomfort, and increased color saturation/visual warping as perceptible effects, but it is not clear how any of these effects relate to improved mental health. Psychedelic microdosing can be understood as an ‘active placebo’, i.e., an intervention without robust direct benefits, but also with some recognizable effects. If this view is correct, improved blinding or a sample without positive expectancy would nullify the benefits of microdosing. Although the results of most studies of microdosing have been negative, the technique may have a positive scientific legacy if it inspires the research community to address its blind spot for blinding integrity.

CGRC relies on binary treatment guess data from patients, but treatment belief is a complex construct that cannot be reduced to a single binary variable. We plan to develop an extension that incorporates guess confidence as weights. In the current work, we focused on introducing the concept of CGR using default kernel density estimate parameters, but in a future publication, we will explore alternative statistical formulations of the CGRC. CGRC can be used to estimate the outcome of AEB by adding a treatment effect and an activated expectancy term. We suspect that unblinding 433 participants was malicious, because the effect sizes were much smaller than the within-subject temporal variability, and because participants reported body/perceptual sensations as their primary cue, in contrast to mental/psychological benefits, which were reported by only 23%.

We have only examined acute and post-acute outcomes in a healthy sample, and we cannot rule out the possibility that microdosing is effective in a clinical population or when co-administered with a behavioral therapy.

Figure 1 shows the computational model of activated expectancy bias, which consists of 2 binary nodes (TRT, PT) with values of placebo/active treatment (PL/AC) and 2 continuous value nodes (TEE and OUT). The model is generated by first generating a value for Treatment, then generating a value for Perceived treatment.

The correct guess rate adjustment estimates the outcome of a perfectly blinded trial based on data from an imperfectly blinded trial by separating the distribution of some outcome measure into two subdistributions, estimating the correct guess rate using kernel density estimation (KDE), and drawing 100 samples from each distribution.

Figure 3 shows the correct guess rate curves of the activated expectancy bias model, which shows that the non-CGR adjusted analysis overestimates the treatment effect by 33%.

Figure 4 shows that the effect of microdosing on the PANAS and mood VAS scales is reduced by 40% after adjusting for activated expectancy, although the effect is still significant. The effect of microdosing on the cognitive performance score is not affected by activated expectancy.

Traditional analysis overestimates the known true treatment effect by 30% when both direct treatment effect and activated expectancy bias are active, while CGR adjusted analysis produces identical results for both analysis in the top two rows.

Table 2 shows that all outcomes that were significant in the traditional analysis became insignificant after CGR adjustment, with the exception of the energy VAS, which remained significant after CGR adjustment.

The activated expectancy bias model parameters are shown in Supplementary table 1, and the summary output is shown in Supplementary table 2.

A systematic review of blinding assessment in randomized controlled trials in schizophrenia and affective disorders 2000-2010 was conducted by Baethge, Assall, O. P., Baldessarini, R. J., and Rosenbaum. The authors recommend that pragmatic research, real-world data, and digital technologies aid the development of psychedelic medicine.

A meta-analysis of blinding in pharmacological trials for chronic pain found that the blind leading the not-so-blind was a common problem. The authors suggest that mobile apps may be a new approach for examining and enhancing placebo effects. Blinded trials taken to the test: An analysis of randomized clinical trials that report tests for the success of blinding. 712 Hemilä, H., Hengartner, M. P., Plöderl, M., Howick, J. H., and James, G. (2015) discuss the importance of effect size and method bias.

Researchers found that active placebo control groups of pharmacological interventions were rarely used but merited serious consideration. They also found that positive expectations predict improved mental-health outcomes linked to psychedelic microdosing. Kirsch, I., Kola, I., Landis, J., Krawczyk, B., Kube, T., Rief, W., Soula, A., Pani, L., Nutt, D., & Erritzoe, D. (2014). Measuring belongingness: The Social Connectedness and the Social Assurance scales. Lund, Vase, Petersen, G. L., Jensen, T. S., & Finnerup, N. B. (2014). Randomised Controlled Trials May Underestimate Drug Effects: Balanced Placebo Trial Design.

A systematic review of research on low dose psychedelics (1955 – 2021) and a meta-analysis of the literature on the subject of microdosing (765 Marschall, J., Fejer, G., Lempe, P., Prochazkova, L., Kuchar, M., Hajkova, K., & van Elk, M. 2021). A systematic review of studies on microdosing psychedelics found that adults who microdose psychedelics report 800 health related motivations and lower levels of anxiety and depression compared to controls.

The 16-Item quick inventory of depressive symptomatology (QIDS), clinician rating (QIDS-C), and self-report (QIDS-SR): A psychometric evaluation in patients with chronic major depression. A systematic review and meta-analysis of the success of blinding in antidepressant RCTs, a self-blinding citizen science study to explore psychedelic microdosing, and a preregistered field and lab-based study of awe and aesthetic experiences are discussed.

Watson, Clark, and Tellegen (1988) developed the PANAS scales, and Winkelman, M. J., & MD, B. S. (2019) published a book on psychedelic medicine.