Question 5: Dealing With Difficult People
5 / 52
This post is a part of the Statistical 52, a weekly series where I write about questions and concepts that I think aspiring statisticians should be comfortable with.
Question
You are working on a team with a doctor. You’ve recently analyzed the primary endpoint and found that it wasn’t statistically significant. The doctor suggests that you redo the analysis with a different subgroup, which you politely reject. In response, the doctor suggests that you “do your job” and “just get a number” that the team can publish. How should you respond?
Discussion
TLDR: Some people may not understand the need for principled statistical analysis, but they might better understand what will happen if you don’t have it.
In school, we often focus on the technical aspects of statistics: knowing assumptions, understanding concepts, and implementing models in code. As a statistician, it’s your job to understand all of this on a deep level. But something that’s often missed in school is that statisticians also need strong communication and collaboration skills to work well on a team. Statistics is infamously opaque and is easily misunderstood even by highly educated people — including doctors.
In light of this, I wanted to mix in some behavioral questions into the Statistical 52 to give you something to prepare against. I received many types of questions during my job interviews, but collaboration questions were always asked.
In the particular scenario I described, a physician collaborator has disrespected you and suggested that you “p-hack”(fishing for favorable p-values with slightly different results).
It is not your fault that the experiment did not yield a significant result, but it is common that statisticians will be asked to redo work so that something can be published. You know that p-hacking is wrong, but others with less statistics exposure may not appreciate why this is the case. You may be viewed as just “the numbers person” who does not really understand the underlying science or medical knowledge that motivated an experiment.
So what do you do?
What not to do
Just to get the obvious out of the way, the wrong thing to do is to get defensive and attack or criticize the physician. You and the physician are on the same team, and it’s in no one’s best interest that there’s beef.
What to do
There’s a lot of different ways that this can be approached diplomatically, so I’ll just lay out some key points that you should hit in your answer if you’re asked this.
1. Focus on a shared goal
As a team, you and the physician are trying to publish the results of your experiment. This is a shared goal. The physician is suggesting something that might push the team closer to this goal, but you know that this actually works against this.
Statistical analyses for primary endpoints are typically pre-planned in advance. This is exactly to prevent people from “moving the goalposts”. If it gets out to a reviewer that you redo the analysis simply because it didn’t produce a specific p-value, then your paper is as good as dead. Even more so if you’re asked to provide your data so that your results are being replicated.
You should explain that the team should still report the null result since it represents the result of the pre-planned analysis. It might still be interesting to result the directionality (i.e. was it positive? negative?) of the estimate. By doing what the physician has asked, you are endangering the project and potential manuscript overall.
2. Be polite, respectful and understanding
Maybe this is obvious, but you should stay level-headed in your response. People may have all sorts of feelings about statistics, but you are the expert on it. You know what needs to be done since it’s your responsibility. You may need to take a moment to collect your thoughts and answer. This can help you avoid reflex defensiveness as well. The situation I described is more relevant in an in-person scenario, but I’ve seen it unfold in emails as well.
Given the pressure to publish, it’s natural that you might be viewed with some frustration if you seem like the person standing in the way of that. But you know that even deeper troubles wait even if you oblige this physician collaborator, so it’s good to be firm and polite. If you don’t do it, then a future statistical reviewer will — and your team will have lost a bunch of time waiting for this to happen.
3. Lean towards non-technical explanations
Even if you refuse, a collaborator may still want to know the statistical reason why you refused. This is where good communication skills really come into play. It’s very easy for statisticians to just jump back to technical definitions about p-values and p-hacking, but you might just lose someone’s attention if you do this.
P-values have a very specific definition, and they will lose this meaning if you repeat the analysis. It would be good to explain that even if you get a “good p-value” with a repeated analysis, it will not be the same as the original p-value you got. Even if it’s good, the process that led to it totally invalidates any planning that came before it. This physician understands it as a requirement to publish, but you know otherwise.
As a last note, I acknowledge that there are so many “what abouts” that can accompany hypothetical situations. These may change the answer, but I hope I’ve laid out some basic ideas that you can fall back on when developing your own answer. Rather than focus on a hypothetical, it would be good to try to think about an actual situation that you ran into.
Were you able to answer it correctly? Was there anything that surprised you? Let me know in the comments! See you in the next question.
📦 Check out my other stuff!
Read through my Statistical Garden on the Very Normal website! This digital garden houses all the knowledge I gained as a biostatistics graduate student. It’ll grow as I learn more, and it’s free for you to look through.
You can support me on Ko-fi! YouTube and Substack are the best and easiest ways to support me, but if you feel like going the extra mile, this would be the place. Always appreciated!