idw – Informationsdienst Wissenschaft
Nachrichten, Termine, Experten
Nachrichten, Termine, Experten
Bonn researchers test the use of various large language models to analyze radiological reports: Artificial intelligence (AI) and above all large language models (LLMs), which also form the basis for ChatGPT, are increasingly in demand in hospitals. However, patient data must always be protected.
Researchers at the University Hospital Bonn (UKB) and the University of Bonn have now been able to show that local LLMs can help structure radiological findings in a privacy-safe manner, with all data remaining at the hospital. They compared various LLMs on public reports without data protection and on data-protected reports. Commercial models that require data transfer to external servers showed no advantage over local, data protection-compliant models. The results have now been published in the journal "Radiology".
Everything has to be in its place. Not only on the operating table or in the office, but also with data. Structured reports, for example, are helpful for doctors as well as for further use in research in databases. Later, such structured data can also be used to train other AI models for image-based diagnosis. In practice, however, reports are usually written in free text form, which complicates further use. This is exactly where the application of AI, more precisely LLMs, comes in.
Open and closed models
LLMs can be divided into two categories: The closed-weights models are the commercial, well-known AI variants that are also used in chatbots such as Chat-GPT. Open-weights models, such as Meta's Llama models, are an option that can be run on internal clinic servers and can even be trained further. When applying these models, all data remain stored locally, which makes the use of open LLMs advantageous in terms of data security. "The problem with commercial, closed models is that in order to use them, you have to transfer the data to external servers, which are often located outside the EU. This is not recommended for patient data," emphasizes Prof. Julian Luetkens, comm. Director of the Clinic for Diagnostic and Interventional Radiology at the UKB.
"But are all LLMs equally suitable for understanding and structuring the medical content of radiological reports? To find out which LLM is suitable for a clinic, we tested various open and closed models," explains Dr. Sebastian Nowak, first and corresponding author of the study and postdoc at the University of Bonn's Clinic for Diagnostic and Interventional Radiology at the UKB. "We were also interested in whether open LLMs can be developed effectively on site in the clinic with just a few already structured reports."
Therefore, the research team carried out an analysis of 17 open and four closed LLMs. All of them analyzed thousands of radiology reports in free text form. Public radiology reports in English, without data protection, were used for the analysis as well as data-protected reports from the UKB in German.
Training makes the difference
The results show that in the case of the reports without data protection, the closed models have no advantage over some of the open LLMs. When applied directly without training, larger, open LLMs were better than smaller, open LLMs. The use of already structured reports as training data for open LLMs led to an effective improvement in the quality of information processing, even with just a few manually prepared reports. The training also reduced the difference in accuracy between large and small LLMs.
"In a training session with over 3,500 structured reports, there was no longer any relevant difference between the largest open LLM and a language model that was 1,200 times smaller," says Nowak. "Overall, it can be concluded that open LLMs can keep up with closed ones and have the advantage of being able to be developed locally in a data protection-safe manner."
This discovery has the potential to unlock clinical databases for comprehensive epidemiological studies and research into diagnostic AI. "Ultimately, this will benefit the patient, all while strictly observing data protection," explains Nowak. "We want to enable other clinics to use our research directly and have therefore published the code and methods for LLM use and training under an open license.
https://github.com/ukb-rad-cfqiai/LLM_based_report_info_extraction/
Funding: This study was supported by the Open Access Publication Fund of the Rheinische Friedrich-Wilhelms-Universität Bonn and by the state of North Rhine-Westphalia (SIM-1-1, Innovative Secure Medical Campus).
Publication: Sebastian Nowak et al: Privacy-ensuring Open-weights Large Language Models Are Competitive with Closed-weights GPT-4o in Extracting Chest Radiography Findings from Free-Text Reports; Radiology; DOI: https://doi.org/10.1148/radiol.240895
Press contact:
Dr. Inka Väth
Deputy Press Officer at the University Hospital Bonn (UKB)
Communications and Media Office at Bonn University Hospital
Phone: (+49) 228 287-10596
E-mail: inka.vaeth@ukbonn.de
About Bonn University Hospital: The UKB treats around 500,000 patients per year, employs around 9,500 staff and has total assets of 1.8 billion euros. In addition to the 3,500 medical and dental students uring the year, 550 people are trained in numerous healthcare professions. The UKB is ranked first among university hospitals (UK) in NRW in the Focus Clinic List, had over 100 million in third-party funding for research in 2023 and has the second-highest case mix index (case severity) in Germany. The F.A.Z. Institute awarded the UKB first place among university hospitals in the category "Germany's Training Champions 2024".
Dr. Sebastian Nowak
Data Scientist, University of Bonn
Clinic for Diagnostic and Interventional Radiology
Bonn University Hospital
E-Mail: Sebastian.Nowak@ukbonn.de
Sebastian Nowak et al: Privacy-ensuring Open-weights Large Language Models Are Competitive with Closed-weights GPT-4o in Extracting Chest Radiography Findings from Free-Text Reports; Radiology; DOI: 10.1148/radiol.240895
https://doi.org/10.1148/radiol.240895 Publication
From text to structured information with AI: Dr. Sebastian Nowak clarifies the data protection-safe ...
Rolf Müller
University Hospital Bonn (UKB)
Criteria of this press release:
Journalists
Information technology, Medicine
transregional, national
Research results, Scientific Publications
English
From text to structured information with AI: Dr. Sebastian Nowak clarifies the data protection-safe ...
Rolf Müller
University Hospital Bonn (UKB)
You can combine search terms with and, or and/or not, e.g. Philo not logy.
You can use brackets to separate combinations from each other, e.g. (Philo not logy) or (Psycho and logy).
Coherent groups of words will be located as complete phrases if you put them into quotation marks, e.g. “Federal Republic of Germany”.
You can also use the advanced search without entering search terms. It will then follow the criteria you have selected (e.g. country or subject area).
If you have not selected any criteria in a given category, the entire category will be searched (e.g. all subject areas or all countries).
