Discussion about this post

User's avatar
Modern Discontent's avatar

Well, I apparently didn't set up Abstract #3 Poll's well. Hopefully no one looked up the answers before giving their response! 😉

The majority of responses for all 3 polls were correct:

1. Generated

2. Original: https://www.nejm.org/doi/10.1056/NEJMoa2212270

3. Generated

I removed the end line from the generated abstracts which included where the original article was. I also removed the line at the end of the conclusion for the original abstract which pointed to a VA clinical trial. This one was a bit more contentious since my prior article mentioned that the clinical trial links for the generated article tended to lead to dead ends, and by removing it in the original I sort of removed that context clue that could have helped.

In general, it appears that ChatGPT can't do well will numbers.

Abstract #1

Note that Abstract #1 uses Confidence Intervals but with percentages. I'm unsure about this method but it's usually the case that CI is given in odds ratios, and that's what the authors of the original list:

"... IRR of individuals who received an mRNA vaccine within 120 days from the last dose was 0.03 (95% CI, 0.03-0.04; P < .001), whereas IRR of individuals who received an adenoviral vector vaccine after 120 days was 0.21 (95% CI, 0.19-0.24; P < .001)."

It could be that ChatGPT did not understand how to process IRR, and so maybe did what it could with percent differences, although remember here that random numbers were used. Also, note here that the original abstract listed different ratios for each vaccine platform, which ChatGPT didn't and appeared to have collected under the umbrella of "vaccines".

Abstract #2

This comes from the NEJM. I took to removing the last sentence which links to a clinical trial:

"(Funded by the Veterans Affairs Cooperative Studies Program; ClinicalTrials.gov number, NCT02185417.)"

What's strange here is that ChatGPT actually gave a completely different conclusion. Where the NEJM article suggests no difference between the two drugs, ChatGPT instead argued that one was more beneficial:

"In this large, randomized, controlled trial, chlorthalidone was superior to hydrochlorothiazide in reducing the incidence of major cardiovascular events in adults with hypertension. These findings support the use of chlorthalidone as a preferred treatment option for hypertension."

Talk about actually giving out bad information! This issue is far more egregious than just making up numbers alone since the fake numbers created a wrong conclusion.

Abstract #3

For this one, the abstract came from an article in Nature. This is why the formatting is in paragraph form, so I hope no one suggested that this was fake due to this format (remember that ChatGPT was told to format the generated abstract in the style of the specified journal).

Here, the abstract constructed is actually close to the original. It's possible that the formatting of Nature helps for ChatGPT to link information together in a more cohesive manner. Take a look at the original and see for yourself if there are egregious differences:

https://www.nature.com/articles/s41591-022-02126-1

"Immune checkpoint inhibitors have shown efficacy against metastatic triple-negative breast cancer (mTNBC) but only for PD-L1positive disease. The randomized, placebo-controlled ALICE trial (NCT03164993) evaluated the addition of atezolizumab (anti-PD-L1) to immune-stimulating chemotherapy in mTNBC. Patients received pegylated liposomal doxorubicin (PLD) and low-dose cyclophosphamide in combination with atezolizumab (atezo-chemo; n = 40) or placebo (placebo-chemo; n = 28). Primary endpoints were descriptive assessment of progression-free survival in the per-protocol population (>3 atezolizumab and >2 PLD doses; n = 59) and safety in the full analysis set (FAS; all patients starting therapy; n = 68). Adverse events leading to drug discontinuation occurred in 18% of patients in the atezo-chemo arm (7/40) and in 7% of patients in the placebo-chemo arm (2/28). Improvement in progression-free survival was indicated in the atezo-chemo arm in the per-protocol population (median 4.3 months versus 3.5 months; hazard ratio (HR) = 0.57; 95% confidence interval (CI) 0.33–0.99; log-rank P = 0.047) and in the FAS (HR = 0.56; 95% CI 0.33–0.95; P = 0.033). A numerical advantage was observed for both the PD-L1positive (n = 27; HR = 0.65; 95% CI 0.27–1.54) and PD-L1negative subgroups (n = 31; HR = 0.57, 95% CI 0.27–1.21). The progression-free proportion after 15 months was 14.7% (5/34; 95% CI 6.4–30.1%) in the atezo-chemo arm versus 0% in the placebo-chemo arm. The addition of atezolizumab to PLD/cyclophosphamide was tolerable with an indication of clinical benefit, and the findings warrant further investigation of PD1/PD-L1 blockers in combination with immunomodulatory chemotherapy."

Expand full comment
Kirsten's avatar

I laughed at your comment about AI characters dissolving into the scenery. 😅 That would be a hurdle to get over.

Thanks for linking to my stack. 💕 I appreciate JJ's streams and perspective; the biology really clears away some significant cobwebs over the truth of these past 3 years, and what might be coming in the future. I get to revisit my love of biology too, which I studied in college.

Expand full comment
10 more comments...

No posts