Open Thread: Studies and why the hell they're hard to read!
A place to rant, ask questions, and have discussion about reading studies and the barriers that come with the attempt.
After my post on Monday on how I tackle studies I stewed for a bit on that topic.
I wouldn’t consider myself anywhere near being a good analyzer of studies, but with at least having formal background and essentially putting my heads into the papers in the past year I figured that I have at least gained some better insights now compared to previously... I hope. 🤷♂️
However, most people may not have the time to read papers, and even if they do they can appear extremely intimidating.
How do you go from having no prior background in paper reading all the way to being expected to read some of the current papers on COVID, any clinical trials, or anything else related to medicine?
It’s an extremely tall order for anyone really.
Although Monday’s post didn’t go super in depth and was more of my perspective, I’m sure a ton of people have questions or a ton of confusion as to how to read studies.
Given the fact that we should be encouraging more people to read studies for themselves, I wanted to see how many issues people have and whether they are rather reticent or intimidated to even try.
So please use this post to let out your frustrations or ask some questions. Is there a study out there that confused you, or any general questions about a study that bothers you?
Or even rant about any issues you have with studies. Is there something about them that grinds your gears? Is the organization too convoluted? Or do you not like how manipulative the presentation of studies can be?
Any thoughts, comments, or questions that come to mind let them out here!
Also, to help out with the conversation here are a few questions to consider. Feel free to answer them or keep them in mind when leaving a comment or question (including if you wanted to respond to any of the polling questions with an explanation).
*This one is something I’ve been considering for some time as a perk for paid members, so it would be interesting to get some perspective from you all and see if this would be something you would find value in.
I’m sure there are plenty of other questions I can ask, but please feel free to ask questions or make comments below! I and everyone else would like to hear what you have to say and get a conversation going!
Edit: I of course made the grave mistake of not including the Materials & Methods section for the 3rd and 4th questions! Please answer the poll question below if you find that the Materials & Methods section is actually the easiest/hardest part of a study! Apologies for overlooking including that section, of course I would overlook it!
If you enjoyed this post and other works please consider supporting me through a paid Substack subscription or through my Ko-fi. Any bit helps, and it encourages independent creators and journalists outside the mainstream.
OF COURSE I forgot to include materials and methods! I'm not sure how to fix this but I may just include another poll question.
The graphs and figures section is also usually pretty hard.
My main personal frustration is this: I have limited time, perhaps 2-4 hours a day. I wan to find something interesting, that I could write about truthfully to inform, entertain and engage my readers.I like articles that could lead to unexpected conclusions.
So I have to read articles, twitter posts, news items etc. When looking at science articles I am not sure if the article has any amazing juicy material worth reporting on. For example (made up)
"Covid-19 incidence trends in prediabetic population of Kansas City, MO"
Such an article may be total dreck, or it may contain explosive findings. When I start the article out, I do not know. The abstract also does not help because it will say that the "vaccine is safe and effective" and abstracts are often highly misleading (even for more boring matters). So I have to look at the article to figure out if I should spend 2, 5, or 30 minutes. The risk is waste of time or missing a super amazing material.
To help myself decide which way to go, I often jump to figures and graphs to see what kind of data it provides and go from there. Any mention of "unvaccinated" is usually a good indicator that the article needs to be explored.
When I write, I always imagine a smart and nasty fact checker standing over my shoulder, ready to notice any mistakes and misrepresent my post by playing the mistakes up. So I am careful to qualify when I am not sure and write as transparently as I can.
Also I am mindful of wasting my readers time. If I waste 1 minute of each person who opens my post, I would end up wasting roughly ten 24-hour person-days total. So I end up deleting a lot of extraneous stuff.
I also give up about 2 out of 3 article ideas because my ideas were wrong, the results may be uninteresting, etc. Also, I try to have one idea, or at most 1.5 ideas in a post to avoid distraction. For example, yesterday, in my comparison of two breast milk studoes, I also wanted to discuss Gorski's critique of the 2022 breast milk study (and I am being generous to Gorski here), but decided against it to avoid idea overloading.
I loved the "How I read articles" post and I think that every substacker reporting on science needs to at least read it.
You and Brian play a very important role here.
Re: press. Sadly, their job is not to inform us, their job is serve their owners and "influence" us as desired, so my expectations are low. I wish that every journalist reads your "how to read articles" post and at least makes an effort to go beyond the headline and the last paragraph of the abstract. A tall order for people who chose journalism due to their inability to do science, I know.