[Opinion] Science and LLMs don't go together well

Two posts on Retraction Watch yesterday highlight yet again how the big publishers in the scientific publishing industry are failing at being gatekeepers of high-quality science, and how many scientists don’t seem to grasp that they should be writing their papers themselves.

“As a nonsense phrase of shady provenance makes the rounds, Elsevier defends its use” (Frederik Joelving, Retraction Watch).

In this first one, a “fingerprint” of automatically-generated papers is described. These fingerprints, like the “tortured phrases” identified by Guillaume Cabanac [arXiv], are basically things that no scientist will ever write and that no diligent human reviewer or editor should let through. Cabanac identifies for instance “profound neural organization” used instead of “deep neural network”, a clear sign that some automated method just tried to find synonyms in an effort to mask plagiarism, without understanding the context.

This time, the fingerprint is “vegetative electron microscope” (or microscopy), which… doesn’t really mean anything? The most likely explanation found for its appearance (in more than 20 published papers) is because of an OCR error: a machine tried to convert a two-column 1959 paper to plain text and accidentally merged the two columns, where “vegetative” and “electron microscopy” appeared next to each other.

But the news here isn’t the fingerprint, but rather the baffling reaction of Elsevier when asked about a 2024 paper published in one of their journal [Elsevier]. To be clear, “vegetative electron microscope” is something that no one has ever written and that doesn’t make sense. But a Elsevier spokesman wrote that “the Editor-in-Chief confirmed that ‘vegetative electron microscopy’ is a way of conveying ‘electron microscopy of vegetative structures’ so he was content with the shortened version to be used.”

This is nonsense. First because, again, no-one uses this “shortcut”. Second, because the thing studied at that point of the paper is bacterial cellulose, which is different from plant cellulose and therefore not even vegetal. The presence of such an obvious mark of automated paper writing should be an instant retraction, and launch an investigation into any previous paper published by the authors.

“As Springer Nature journal clears AI papers, one university’s retractions rise drastically” (Avery Orrall, Retraction Watch)

Writing a scientific paper with ChatGPT or other LLMs is a terrible idea. Using AI assistance to write lowers our critical thinking abilities, as even Microsoft realises [Microsoft Research].

Convincing scientists, particularly those under heavy pressure to publish a lot, not to use these tools is a tall order. The bare minimum, which the Springer Nature publisher requires, is to disclose the use of an LLM. More than one hundred papers that failed this bare minimum requirements have been retracted so far by the Neurosurgical Review journal. At least in this case the journal seems to be doing its job, but I find the response from one of the retracted authors [Sivakamavalli Jeyachandran via RetractionWatch], who disagrees with the retraction, really disheartening.

His argument is:

• Lack of standard for detecting LLM-written content makes the retractions arbitrary. That’s true, but “it’s hard to detect fraud” is a poor argument for “not acting when fraud is detected”.
• It was a letter or commentary, not a full research paper, so it doesn’t really matter. If it matters enough to be published, it matters enough to be retracted – in my opinion.
• Retractions should only be for data fabrication, plagiarism or breaches of journal policy. Well, this was a breach of journal policy, so I’m not sure how that’s an argument against retraction. Also, LLM output is arguably plagiarism by essence, but that’s another debate…

He at some points refers to his undisclosed usage of an LLM as a “minor technical infraction”. I certainly disagree – and am glad that Springer seems to disagree as well.