Demystification – a solution for assessment of real-world effectiveness?

Nearly 100 years ago Archie Cochrane and Austin Bradford Hill proposed a three-dimensional assessment for evaluating clinical interventions: Can it Work? Does it work? Is it Worth it? [2, S1]. Table 1 proposes a strategy to answer these questions by description of three outcome dimensions that are assessed under two different conditions from three different perspectives. The strategy requires a different architecture of studies (form and function) and different tools.

Nearly 100 years ago Archie Cochrane and Austin Bradford Hill proposed a three-dimensional assessment for evaluating clinical interventions: Can it Work? Does it work? Is it Worth it? [2, S1]. Table  1 proposes a strategy to answer these questions by description of three outcome dimensions that are assessed under two different conditions from three different perspectives. The strategy requires a different architecture of studies (form and function) and different tools.

Even randomized controlled trials (RCTs) have limitations
Although RCTs are considered the Gold Standard in clinical research several limitations are often overlooked. We consider four examples. The RCT is expected to distribute known and unknown confounders equally among study groups. This assumption is justified in case of 'adequate trial size' . We estimated the minimal necessary size of RCTs that can guarantee the equal distribution of confounders based on four assumptions: 10 confounders, independent from each other, dichotomous distribution, and maximal tolerated difference of 5%. The result of this model revealed a minimal number of 1000 participants per trial to guarantee the fairly equal distribution of confounders [S4, S5]. Most studies that support clinical recommendations include less than 1000 patients (Appendix II).

Demystification -a solution for assessment of real-world effectiveness?
Second, open (unblinded; unmasked) studies are biased by the strong and weak expectations of both patients and attending practitioners. Unmet strong expectations result in refusal of participation and contribute to sampling bias. Patient expectations were classified as 'weak' if their expectations are not met, but the patient nevertheless participates in a study. It is shown in Appendix III that large difference in the expectations towards the alternative treatment arms (e.g. 80% of patients prefer intervention ' A' and 20% 'B') predict that the treatment group with the higher preference rate will achieve better study results [S6-S10]. Third, de-masking or de-blending during the course or the study may cause bias. Several examples demonstrate the effects of deblending [S11,S12]. Recently a meta-meta-analysis claimed that blinding has no effect on the results of RCTs [S13], yet a simple selection bias should be excluded, which may explain the reported effect [S14]. Fourth, non-inferiority designs can only be used to compare interventions with previously confirmed benefit [S15]. Noninferiority is difficult to interpret unless we have evidence that both the experimental group and the comparator are superior to placebo [S16].

The RWC: a 'Natural Chaos' from the perspective of the investigator
In clinical practice, decisions are supported either by external evidence (based on published data), internal evidence (based on personal experience) or both. As an example of 'Natural Chaos' , we compared 330 recommendations of international guidelines from 11 countries and observed congruence in recommendations i.e. at least 66% congruency in 50 of 330 (15%) recommendations, incongruence in 213 (65%), and undetermined recommendations in 67 (20%). One of the most likely interpretations is that clinical experts who review these guidelines are exposed to two different types of information. When treating patients, they observe effectiveness data under RWC. Scientific publications describe efficacy data generated under ESC. It is likely that both efficacy and effectiveness data influence the opinions of decision makers differently [S17-S19] and the different information may lead to the observed incongruence. Our 42 tables summarizing this literature are available upon request.

The solution of the unsolved problems
RCTs can address the "Can it work? question while neglecting evidence about effectiveness and value. The appropriate tool for assessment of RWE and Value, the Pragmatic Controlled Trial [4,5,S20-S22] is based on four principles. First, the practitioner and patient but not the investigator decide about the allocation. Second, the investigator defines which artificial intelligence tools should be applied to render the 'Natural Chaos' evaluable under RWC. Third, these tools, e.g. Bayes Theorem [S23] are used to stratify the real-world data for evaluation. Fourth, allocation and evaluation must be independent from each other.
This solution classifies all patients with a comparable clinical problem (e.g. breast cancer) who are treated individually according to their risks profiles each related to three different endpoints: main goals e.g. mortality, main treatment side effects e.g. allergic reactions, and total costs of treatment (Appendix IV). The risk profiles of the groups had to be predefined before start of the trial. All study participants are allocated to a high, intermediate or low risk group separately for each endpoint by using a separate algorithm for classification of the risk groups for each of the assessed endpoints. The steering group selects the interventions that will be compared with each other and with a mixed group of the remaining 'any other interventions' . Corrections for multiple testing are necessary.
All patients -without exception -who meet the inclusion criteria e.g. breast cancer and ask for help at a participating center within a defined time window are included in a Pragmatic Controlled Trial (PCT). This procedure meets four essential requirements: A) the practitioner and patient decide about the allocation as requested for pragmatic trials [3,S2,S3]. B) the investigator defines the rules that enable the unbiased evaluation of the 'Natural Chaos' according to the Bayes Theorem [S23]. C) patient self-selection defines the membership to the real-world patient population. D) the trial includes a well-defined control group i.e. the mixed but risk-stratified intervention groups that were not selected as specific intervention target groups.

Research progress and advantages
The goal of this strategy is to relate outcomes to baseline risks. It differentiates three outcome dimensions and reduces the number of human experiments to an absolute minimum. Two or three high quality RCTs will be sufficient for demonstration of the Proof of Principle, or "Can it work". Ethical conflicts associated with unnecessary human experiments, the high costs and complexity of conducting randomized trials, and bureaucratic burden will be reduced. We need bold new approaches to complete a paradigm shift in healthcare from a monodimensional to the three-dimensional assessment of outcomes that was suggested almost 100 years ago by Sir Archie Cochrane and Sir Austin Bradford Hill [2,S1].
Second, the new tool was used to supplement the missing part in the three-dimensional Cochrane-Hill strategy. The work resulted from the collaboration of a large group of American, Brazilian, Italian, Norwegian, and German colleagues and doctoral students. Copyright: ©2020 Porzsolt F. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.