Trial exclusion criteria and their impact on the estimation of antipsychotic drugs effect: A case study using the SOHO database

Abstract

Objectives

To explore the impact upon estimation of drug effect as a result of applying exclusion criteria in randomized-controlled trials (RCT) measuring the efficacy of antipsychotics (AP) in schizophrenia.

Methods

Three characteristics which may act as effect-modifiers of AP, while also common exclusion criteria in RCTs, were identified through literature review: schizophrenia duration, substance use disorder and poor adherence. The SOHO cohort was used to estimate the effect of initiating antipsychotic drugs “A”, “B” or “C” (pooled) upon symptom evolution at 3 months from baseline (CGI-S scale). “Estimated effectiveness” and “estimated efficacy” were drawn from the “SOHO” and “RCT-like” (patients with none of the above-listed exclusion criteria) samples, respectively. Effect-modification and impact of each exclusion criterion on AP effect estimates were explored using non-adjusted statistics.

Results

The “SOHO sample” included 8250 patients initiating drug A, B or C at baseline, whose AP “estimated effectiveness” was ΔCGI-S = − 0.78 (95% CI = − 0.80, − 0.76). The “RCT-like” sub-sample included 5348 (65%) patients whose AP “estimated efficacy” was ΔCGI-S = − 0.73 (95% CI = − 0.75, − 0.70). Patients with short illness duration (≤ 3 years since first AP; n = 2436) experienced significant symptom improvement (ΔCGI-S = − 0.89; 95%CI = − 0.93, − 0.85) compared to patients with duration > 3 years (mean ΔCGI-S = − 0.73; 95%CI = − 0.76, − 0.71). Excluding patients with short illness duration led to a change in AP effect estimates but this was not the case for substance use disorder or poor adherence.

Conclusion

Using certain exclusion criteria in RCTs may impact the drug's effect estimate, particularly when exclusion criteria are AP effect-modifiers representing frequent characteristics among patients with schizophrenia.

Introduction

Randomized controlled trials (RCTs) are designed to measure the pharmacological effect of drugs in humans (“efficacy”) while attributing the observed effect to the drug itself, and not to confounders (“internal validity”). Wherefore, RCT design requires specific features to replicate an experimental setting (Schwartz and Lellouch, 2009), including, but not restricted to, randomization, blinding, or medication adherence monitoring during the trial. Moreover, RCTs are usually performed in highly-selected patient populations (Stroup et al., 2006, Van Spall et al., 2007), recruited in public hospitals – rarely in private practices – and screened using many inclusion and exclusion criteria. In consequence of the selection and homogeneity of clinical settings, physicians and participants, the RCT population often represents a subset of the overall population administered the drug in routine clinical practice and results may not be reproducible across patients managed outside clinical trials (Rothwell, 2005). It is increasingly recognized that controlled experimental settings may jeopardize the generalizability of RCT results, which may not reflect the effect of drugs as prescribed in routine clinical practice, known as “effectiveness” (Eichler et al., 2011). Discrepancy in evidence between RCTs and real-life studies is referred to as the “efficacy-effectiveness gap” (Lehman et al., 1995, Nordon et al., 2016).

RCTs however do not systematically provide poorly generalizable results and the efficacy-effectiveness gap is not paramount. Actually, efficacy-effectiveness gaps may occur under specific circumstances, previously investigated and explored in detail. For instance, Naudet et al. (2011) have explored the association between RCT design features and the effect size of selective serotonin reuptake inhibitors (SSRIs) antidepressants, in patients with major depressive disorders. Results suggest that the effect size of SSRIs is likely greater in non-blinded compared to double-blinded studies, independent of other design features and patients or illness characteristics. The impact of blinding upon the SSRIs effect size was confirmed by Chassang et al. (2015) who showed that, in double-blinded RCTs, the response to SSRIs tends to be smaller than in open-label studies because blinding prevents the effect of “patients' beliefs” from underpinning the positive expectations of the active treatment, inducing a change in patient behavior or the way these perceive their condition, which can result in symptom improvement. More importantly, the authors evoke a possible difference in treatment effect size because of an interaction between “patients' beliefs” and the drug effect. In other words, it is because “patients' beliefs” is an effect-modifier of antidepressants that blinding or not, will impact antidepressants effect estimates. The influence of key characteristics (“drivers of effectiveness”) modifying the effect of drugs upon the risk for efficacy-effectiveness gap was also evoked by other authors (Longford, 1999, Weiss et al., 2012, Huybrechts et al., 2010).

In schizophrenia research, most RCTs performed on antipsychotic drugs include highly-selected patients (Robinson et al., 1996, Hulihan et al., 2013), particularly pre-authorization RCTs, which stresses the importance of evaluating the AP effectiveness, otherwise unknown at launch (Stroup et al., 2006). Hulihan et al. (2013) provided the example for long-acting injectable antipsychotics, for which any advantage over oral short-acting antipsychotics is expected to be related to better medication adherence. Double-blinded RCTs cannot unfold this effect of improved adherence unlike effectiveness studies, where patients while possibly less adherent are also less frequently monitored and overall less inclined to adhere to therapy (Grimaldi-Bensouda et al., 2012). Better quantification of antipsychotics effectiveness using pre-authorization RCTs is an issue of upmost importance considering the dramatic increase in antipsychotics use in recent years (Verdoux et al., 2010).

The present case study was conducted within the realm of the European GetReal Consortium (Innovative Medicines Initiative, 2013), a non-competitive, public-private funded research project aiming to provide new and robust methods for real-world evidence to be generated earlier during drug development. One important objective is to develop guidance on improving the design of RCTs. In this case study, we aimed to: (1) identify effect-modifiers for antipsychotics among patient-related or illness-related characteristics frequently used as exclusion criteria in RCTs; and (2) quantify the impact of applying these exclusion criteria on the estimation of antipsychotics effect. In other words, our work intends to support clinical trials design by providing evidence that exclusion criteria – not mandated by safety – need to be carefully chosen so as to minimize the risk of an efficacy-effectiveness gap.