Evidenced Based Medicine (EBM): Beyond the dogma

A disclosure on reading EBM based guidelines (and interpreting statistical analysis)

Before presenting my comments on 2015 ALS Guidelines I wanna share my thoughts on EBM based guidelines and interpretation of statistical analysis as a “disclosure” for all MEDEST followers and to clarify some concepts on this two methodological approach.

Evidence Based Medicine External Validation and applicability.

EBM is based on RCTs (randomized and controlled) studies as maximum expression of quality of evidences.

The original spirit of EBM was to improve the quality of care for real patients in the real world (external validation). RCT studies are mostly based on controlled group of patients and regional organizations, expressions of local contexts and not always applicable to a more wide population of patients.

So in the years the concern about GRADE score of evidence (where RCTs trials are the highest expressions of evidence), made EBM based guidelines more focused on internal validation than external validation and applicability in widest clinical contexts.

Everyone of us when comes to clinical practice have to consider this potential bias.

Local context, individual clinical experience and local experts opinion can be the bridge between internal and external validation of RCT studies and EBM based guidelines.

Statistical Analysis

Similar considerations can be done on statistical analysis and statistical significative results.

To better explain this concept consider the result of this trial, Therapeutic Hypothermia after Out-of-Hospital Cardiac Arrest in Children, recently published in NEJM.

Results: The proportion of survivors with VABS-II scores of 70 or more at 12 months was not significantly different between the two groups (20% in the hypothermia group vs. 12% in the normothermia group; relative likelihood, 1.54; 95% confidence interval [CI], 0.86 to 2.76; P=0.14)

Authors conclusions: In conclusion, in comatose children who survive of out-of-hospital cardiac arrest, therapeutic hypothermia, as compared with therapeutic normothermia, did not confer a significant benefit with respect to survival with good functional outcome at 1 year. Survival at 12 months did not differ significantly between the treatment groups.

The authors conclusions are based on a P value of 0,14 that effectively is not relevant froma a statistical point of view. But what about ethic and clinical side of the picture? Can we ignore such a numerical difference on the base of a statistical interpretation?

In an other article (see in the references Difficulty interpreting the results of some trials: the case of therapeutic hypothermia after pediatric cardiac arrest.)  is well illustrated this dilemma, simulating a conversation between a physician and a parent of a post cardiac arrest comatose child.

From the article:

Doctor: I’m really sorry, but your child may have serious brain damage as a result of his cardiac arrest.

Parent: That’s terrible! Isn’t there anything we can do?

Doctor: I’m afraid not. There are some interventions that have been suggested, but they’ve not been shown to be effective.

Parent: What interventions?

Doctor: Well, cooling the body for a couple of days, for example. It’s been tried in neonates with birth asphyxia and adults after cardiac arrest.

Parent: But … if this intervention is used in neonates and adults, how can you say it won’t work in children?

Doctor: Well, in a recent study including almost 300 children, 20 % of those who were cooled survived with good brain function versus just 12 % of those who weren’t cooled. Neurological status improved in 38 % of the cooled children compared with only 29 % of the non-cooled. And, 28 days after the arrest, the mortality rate was 10 % lower in cooled children (57 % versus 67 %). Unfortunately, when the researchers applied the standard statistical rules that we use to interpret all scientific research, there was more than a 10 % possibility that these differences were due to chance, so we can’t recommend it.

Parent: But those results are really encouraging. Even if statistics tell you that this may be due to chance, there’s still the possibility that it wasn’t and I’d like my child to have that opportunity. Maybe the treatment’s expensive?

Doctor: No, that’s not the issue.

Parent: Was it dangerous then?

Doctor: Quite safe actually. Potassium and platelet levels went down a little, but with no harmful consequences. There is a risk that the heart rhythm can be affected; some of these abnormalities can even be quite dangerous. In the same study, serious abnormalities of the heart rhythm occurred in 11 % of the cooled children and 9 % of the others. Reduction in body temperature also increases the risk of infections; the investigators of this study reported that 46 % of cooled children developed an infection, compared with 39 % of the other children.

Parent: So, the treatment is associated with some risk but can still improve the chances of my child surviving… how can you balance the benefits and the risks for my boy?

Doctor: Honestly, I don’t know. If I just have to use numbers… 12 children would need to be cooled instead of kept at normal temperature in order to have one additional child with a good clinical outcome. And, 15 children would need to be cooled for one child to develop an infection.

Parent: Please, try this treatment on my child.

Statistical analysis is not the only determinant in daily clinical practice such as in real life. Reading the results of clinical trials beyond statistical analysis is important when we arrive to apply those results in our clinical practice.

Again, clinical gestalt and local experiences has to be considered when interpreting statistical analysis of clinical trials.

References

• Ana Fernandez et al. Evidence-based medicine: is it a bridge too far? Health Research Policy and Systems201513:66 DOI: 10.1186/s12961-015-0057-0

• Jean-Louis Vincent and Fabio S. Taccone Difficulty interpreting the results of some trials: the case of therapeutic hypothermia after pediatric cardiac arrest.
  Critical Care201519:391 DOI: 10.1186/s13054-015-1121-4