AI’s potential and the risks for research
There is no doubt that artificial intelligence (AI) will play an ever greater role in our daily lives. Designed to emulate human cognitive abilities, AI can easily outperform us at certain tasks, completing in seconds what might take a person days, weeks, or even months.
Beyond automating tasks, its capacity to keep learning allows it to evolve and expand into new areas. We can see this in image-generating AI, which produces remarkably realistic illustrations, and in tools designed for communication, such as ChatGPT.
AI can offer many benefits to humanity, but indiscriminate use can also have catastrophic consequences. In this article, I explore the dangers associated with using AI in scientific research, particularly in health.
My experience with ChatGPT
My desire to understand technological advances led me into the world of AI chatbots. A colleague first introduced me to them and warned me about the risks they could pose to the future of scientific research, particularly plagiarism, as many students had begun using them to write academic assignments (YANG, 2023).
His concern was that AI-generated texts draw on information available online without properly citing the material used. This made it impossible to identify the original sources or determine whether passages had been plagiarized. He wondered: “Could existing plagiarism detection tools identify these patchwork texts?” However, despite those concerns, I initially doubted that any technology could write so convincingly. To my surprise, it was more than possible.
I wanted to learn more about AI chatbots and was struck by their writing ability. They could imitate scientific writing so closely that a reader might not be able to tell whether a text had been written by a person or a machine. Fascinated, I began exploring ChatGPT and soon discovered both its potential and its serious dangers.
When the data and sources do not exist
From the outset, I wondered where all that information came from. When I asked it to create a sample article and then asked which sources it had used, the answer was always the same: “As a language model, I do not have access to the specific list of sources.” This brought my colleague’s warning back to mind. What began as a nagging doubt soon became a problem too big to ignore.
To explore how it might support research, I asked the chatbot for up-to-date figures on nursing students’ participation in research projects and groups. It listed the studies and resources I could supposedly consult: “Pesquisa Nacional de Perfil dos Graduandos em Enfermagem (2018)”, “Censo da Educação Superior (INEP)”, and “Cadastro Nacional de Grupos de Pesquisa (CNPq)”. At first glance, nothing seemed suspicious.
I then asked for the specific figures. That was when I received the following response:
“Excellent!” I thought. “These are exactly the figures I need for my research, and I haven’t been able to find them anywhere.” I did what any researcher should do: I searched Google for the original studies so I could read them. Nothing came up. I asked the chatbot for more information, from the exact study title to a link, but neither led anywhere.
That was when I encountered something deeply concerning: every link the chatbot provided was either broken or led to an unrelated article or webpage. Even the article titles produced no results when I searched for them on Google. I decided to ask directly whether it was making up the data. This was its response:
I knew that the study “Perfil da Enfermagem no Brasil” did exist, so I began working backward to trace the information the chatbot had given me. However, the only study available on the website of Brazil’s Federal Nursing Council, COFEN (http://www.cofen.gov.br/perfilenfermagem/index.html) was not from 2018 and contained no data on nursing students’ participation in research projects during their undergraduate studies. Nor could I find any studies citing the supposed document.
The detailed response might make you wonder whether the study had once existed but had since been removed from the internet. I therefore contacted the researchers responsible for the real study to ask about the information. By email, they sent me every publication arising from the original research, along with the final report and the datasets available on COFEN’s website. After extensive reading, I found none of the AI-generated information in any of the materials. This strongly suggested that the chatbot had fabricated the data.
Fabricated references that look real
Every reference list I requested contained fabricated studies, without exception. The chatbot attributed them to real authors with expertise in the relevant field, even though those authors had never conducted the studies. In other cases, it invented titles or combined the titles of two or more studies into a new one. Yet the references looked convincing at first glance: the chatbot formatted them correctly and supplied a journal name, volume, page numbers, and even a Digital Object Identifier (DOI), which invariably turned out to be fake or to belong to a different study.
This may seem like an isolated experience, but in a recent Nature news article about ChatGPT, Brian Owens raised concerns about chatbot reliability, citing accounts of these tools producing entirely fictitious reading lists (OWENS, 2023). His report echoed an earlier article on Nature’s website that criticized listing ChatGPT as a coauthor of scientific papers and explained the structural and ethical problems involved (STOKEL-WALKER, 2023).
The consequences for health research
Studies retracted for misconduct can continue to be cited long afterward (OLIVEIRA, 2015). Against that background, these weaknesses point to a dangerous future for health research: fabricated information can enter the literature, cause harm on an immeasurable scale, and persist even when retractions fail to reach the wider public.
Scientific evidence underpins treatments and public health policy. Its indiscriminate or malicious use can therefore cause lasting harm, undermining trust in professional practice, collective professional advocacy, and the identities of professionals who have chosen, for now, to replace their role as researchers with ready-made AI answers.
How to use these tools with caution
However serious these problems are, they are common in AI systems that are still learning. ChatGPT, for example, was launched in November 2022, and its AI is still developing. It may be able to address its current shortcomings in the near future. Despite the weaknesses described here, continued use could help it become a valuable tool for educators and researchers.
For students, my advice is to wait before venturing into the world of AI chatbots: distinguishing false information from accurate information requires the right academic skills. Educators and researchers should proceed with caution, uphold high standards, and always question the information these tools provide. AI chatbots such as ChatGPT are nevertheless likely to play a fundamental role in scientific research, becoming part of researchers’ everyday work in the not-too-distant future and helping strengthen the links between teaching, research, and community engagement.
References
OLIVEIRA, M.B. A epidemia de más condutas na ciência: o fracasso do tratamento moralizador. Scientiae Studia, São Paulo, vol. 13, no. 4, pp. 867–897, 2015. DOI: 10.1590/S1678-31662015000400007. Available at: https://www.scielo.br/j/ss/a/hMs7Y4FVZvbjwHkmYKTn3Ny/?lang=pt#. Accessed March 11, 2023.
OWENS, B. How Nature readers are using ChatGPT. Nature, London, vol. 615, p. 20, February 20, 2023. DOI: 10.1038/d41586-023-00500-8. Available at: https://www.nature.com/articles/d41586-023-00500-8. Accessed March 11, 2023.
STOKEL-WALKER, C. ChatGPT listed as author on research papers: many scientists disapprove. Nature, London, vol. 613, pp. 620–621, January 18, 2023. DOI: 10.1038/d41586-023-00107-z. Available at: https://www.nature.com/articles/d41586-023-00107-z. Accessed March 11, 2023.
YANG, M. New York City schools ban AI chatbot that writes essays and answers prompts. The Guardian, New York, January 6, 2023. Available at: https://www.theguardian.com/us-news/2023/jan/06/new-york-city-schools-ban-ai-chatbot-chatgpt. Accessed March 11, 2023.