The origin of the previous posts, though they were lengthy and detailed in some areas, was triggered by reading about issues with reporting on results of medically-related studies. Specifically, it relates to how results of trials are translated to approving agencies and to the public. This is not about misrepresentation related to recent popular discussion of “bias”. This is misrepresentation by use of invalid assumptions or by deliberate misstatement of results.
Percentages
Suppose that you and I decide to have a contest about our weight. You now weigh 206 lbs. I now weigh 212 lbs. Over the Holidays, since we’ve splurged on rich, delicious foods, we’ve both gained weight. Now it’s the New Year and we want to go back to our previous weights – or close to them. Over the next two months, we will work to reduce our weights. Whoever loses the most in percentage of current weight, wins $50 from the other. It’s not much of a prize but because of our competitive natures, we are both motivated to win!
For review, what would be a measure of losing a percentage of our current weight?
Percentage of Weight Lost = (Lbs lost) / (Original Weight) X 100%
So, if you lost 10 lbs, you would have lost 10 / 206 X 100% = 4.9%. If I lost 10 lbs, I would have lost 10 / 212 X 100% = 4.7%. So, when losing the same number of pounds, we lose different percentages because the base weights are different. Similarly, if we both lost 5%, for example, you would lose 10.3 lbs and I would lose 10.6 lbs.
That seems simple enough.
At the end of the contest, you weigh 198 lbs and I weigh 200 lbs. Who won?
- You lost 8 lbs. => 8 / 206 X 100% = 3.9%.
- I lost 12 lbs => 12 / 212 X 100% = 5.7%
Those numbers show who won and by what amount. There are at least two simple ways we can tell our friends about the results:
- I beat you by 1.8%. (5.7 – 3.9 = 1.8). That doesn’t sound like much.
- I beat you by 46%! Now, that sounds like much more! I tell that story by this arithmetic: 1.8% / 3.9% => 46%
If you study a set of trial data (not just for pharmaceuticals) and the results of the comparison between two subjects is only 1.8%, you may want instead to report those results in a way that shows a higher number to paint your study in a better light. Though both answers have merit (i.e., both 1.8% and 46% are arithmetically correct), understanding the basis of the comparison is important to know how to understand results.
Probability
For probability calculations, it’s important to understand the underlying mathematical requirements to properly calculate results. Incorrect calculations can result in incorrect results in any areas of math. Assuming the math is correct and the assumptions are valid, probability misuse may be more likely in the blurring of terminology that’s related. This means mixed use of reference to “probability”, “chance”, and “odds”.
Chance is a vague term and I won’t spend much time on it here except as a reminder that we use it in everyday conversation in ways where we may actually mean (in technical terms) either probability or odds.
Remember that probability is a number between 0 and 1. It represents the likelihood of an event and is calculated as:
Probability = (Expected outcomes) / (Number of possible events)
The probability of rolling a 5 when rolling one six-sided die is: 1 [there is only one possible “5”] / 6 [there are size possible results] = 1/6 or 0.17 => 17%.
The odds, however, are not the same as the probability. Odds are calculated as:
Odds = (Probability an event occurs) / (Probability the event does not occur)
The odds of rolling a “5” in the example above are: 0.17 / .083 = 0.20. The numbers are close but they have different meanings. We are more likely to think about odds when both the probability and the odds are higher.
If the probability of an event is 0.80, the probability that the event does not occur is 0.20. The odds then are: 0.80 / 0.20 = 4. We normally hear this as 4:1.
So, remember that “probability” or “odds” are related but are not the same.
Statistics
Statistics raises more likely challenges from the math theory than we will cover here. Statistics involves making assumptions that may be significantly flawed or, as with percentages, misrepresented. In addition, if you follow statistics (or probability) you may know about the debate over frequentist methods vs Bayesian methods. (Here is a simple explanation). Perhaps at the academic level we can debate which of the statistical schools of thought is “correct”. Both methods have a foundation. Some argue that the concept of “p-value” is a serious flaw in frequentist methods. Bayesian methods have become more common in recent years and in some circles are considered superior.
For our purposes, because of the complications of trying to clearly explain either without lengthy discussion, we leave the question of misrepresentations of statistics at this:
- Statistical calculations and assumptions are more complicated than simple percentages or probabilities. This results in the need for more understanding of the methodology (whether frequentist or Bayesian).
- The assumptions of a study are critical to understand. Since the goal of using statistics is often to reach a conclusion or stimulate an action, the assumptions in the related study must be valid. It’s human nature to want to get a “good” result (whatever “good” means to you or for the study).
- There is a cartoon by Sidney Harris showing two professors at a board discussing a mathematical proof/formula. In the middle of the proof is the phrase “Then a miracle occurs”. The reviewed says that step should be a little more explicit. The same can happen in statistics. A study can make a leap where an implied “miracle” occurs to move from a set of data or results to a conclusion. Conclusions using statistics should have solid logic (along with defined assumptions) to reach a conclusion.
Conclusion
Many of us don’t notice the types of issues mentioned here. They can affect you either directly or indirectly. Directly, if you make a life or health decision based on misrepresented results of a study or trial. Indirectly, if someone else or a group makes decisions on invalid results.
Disclaimer: this post does not contain recommendations about treatment of medical conditions. The following is only a review of the numbers reported in trial results as published by others. Seek advice from a medical professional concerning your own health issues.
Two Final Examples from Published Results
Lipitor is a commonly prescribed statin intended for the use in reduction of cholesterol. In the insert that accompanies the drug, study results are stated as the following (in part):
“There were no significant differences between the treatment groups for all-cause mortality: 216 (9.1%) in the Lipitor 80 mg/day group vs. 211 (8.9%) in the placebo group. The proportions of subjects who experienced cardiovascular death were numerically smaller in the Lipitor 80 mg group (3.3%) than in the placebo group (4.1%). The proportions of subjects who experienced non-cardiovascular death were numerically larger in the Lipitor 80 mg group (5.0%) than in the placebo group (4.0%).”
If we are noticing uses of numbers, as we’ve discussed here, we see a couple of important points in this section:
- All-cause mortality – means death from any cause not just related to heart health and cholesterol.
- All-cause mortality results in this study were 0.2% less in the placebo group. Not a large number but less than that for the Lipitor 80 group. For those of us who don’t think small percentages are relevant, this may not seem like a relevant difference.
- Cardiovascular deaths reported were 3.3% for the Lipitor 80 group and 4.1% for the placebo group. That’s a 0.8% difference. Less than 1%.
- For whatever reason, non-cardiovascular deaths were larger in the Lipitor 80 group (5.0% vs 4.0%). Larger – but not much larger.
In a separate publication (linked below), these results were mentioned:
“Although the reduction of fatal and non-fatal strokes did not reach a pre-defined significance level (p=0.01), a favorable trend was observed with a 26% relative risk reduction (incidences of 1.7% for LIPITOR and 2.3% for placebo).” From this source.
From our discussion about statistics and percentages, this highlights two things (shown as bold above but not in the original):
- Notice the reference to “significance level”. We have not discussed that in detail but recognize that it is one of the elements of frequentist statistics. Separate from the debate about the validity of “p-values”, this result says the study did not reach the defined significance level.
- Here they show both the relative risk and the actual percentages. The relative risk of 26% is calculate like this: 2.3% – 1.7% = 0.6%; 0.6% / 2.3% = 0.26 => 26%. As with our weight loss example above where we calculated two results, presenting the improvements from Lipitor as a 26% reduction appears more position than the actual difference of 0.6%.
For those who are interested in this, I hope this has helped show where these three categories – percentages, probability, and statistics – can go off track.