idw – Informationsdienst Wissenschaft
Nachrichten, Termine, Experten
Nachrichten, Termine, Experten
Researchers at TU Berlin tested the ability of ChatGPT to supply correct information on climate-related topics. They found out that the language model’s responses are largely correct, but contain errors in the details.
Since it was launched in November 2022, ChatGPT has attracted more than 100 million users all over the world. AI language models, which like ChatGPT are based on machine learning and big data, work with predictions of probability. On demand they generate a response by searching through large quantities of text in the training data for word-patterns relating to the inquiry and decide, with the aid of probability distributions, which word is the next word in a sentence. That may sound pretty easy at first, but in actual fact ChatGPT’s logic is essentially a black box – not even the developers themselves can really say how the AI model comes up with a certain answer. Aside from that, ChatGPT has a tendency to make meaningless guesses rather than rejecting unanswerable queries. Since, in relation to climate change, not only science-based information but also false information circulates in public discourse and the media, scientists from TU Berlin and Berliner Hochschule für Technik wanted to find out how competent ChatGPT is in answering climate-related queries correctly.
Balanced and nuanced arguments
The researchers from the Green Consumption Assistant project, which is developing an AI-based assistant that supports consumers in making more sustainable choices when shopping online, collected 95 questions concerning climate change and put them to ChatGPT. The researchers then evaluated the responses on the basis of the criteria of accuracy, relevance and consistency. The quality of the responses was evaluated by the team, which consulted publicly accessible and reliable sources of information on climate change, such as the current report of the Intergovernmental Panel on Climate Change (IPCC).
“We observed that ChatGPT provides balanced and nuanced arguments, and many responses end with a comment encouraging critical appraisal in order to avoid biased responses,” says Dr. Maike Gossen from TU Berlin. For example, when asked “How is marine life influenced by climate change and how can negative influences be reduced?” ChatGPT mentioned in its response not only the reduction of greenhouse gases, but also reducing non-climatic effects of human activities like overfishing and pollution.
Overall, the quality of ChatGPT responses to the climate-related queries was high. As for inaccurately answered questions, the most common error was caused by the so-called hallucination of facts, i.e. asserting facts that cannot be verified by any sources, or even making completely invented assertions from completely invented sources. For instance, the ChatGPT response to the question “What percentage of recyclable waste is actually recycled by Germany?” was broadly correct, but wrong in the details. In some cases, ChatGPT generated false or fake information like made-up references or fake DOI or URL links. Other errors arose in cases where ChatGPT cited concrete and correct scientific sources or literature, but drew false conclusions from them.
Responses reflect societal misunderstandings
The researchers were also able to observe what is already known, namely that inaccurate responses by ChatGPT often have a plausible-sounding tone and can therefore be falsely perceived as correct. “Since text generators like ChatGPT are trained to give responses that sound right to people, the self-assured response style can lead people to believe that the response is correct,” Dr. Maike Gossen notes. Furthermore, the team encountered deep-rooted prejudices in big language models. Some of ChatGPT’s false responses thus reflect societal misunderstandings about effective measures against climate change, such as the overvaluation of individual behavioral changes and single actions with little impact, at the expense of structural and collective changes with greater impact. Sometimes the responses also seem excessively optimistic with regard to technological solutions as the main way to contain climate change.
Nevertheless, language models have the potential to improve the way information on climate change is communicated, the researchers say. Their ability to process and analyze large amounts of data, and to provide easy-to-understand answers to everyday questions, can make them a valuable source of information on climate change. At the same time, however, there is the danger of language models spreading false information about climate change and promoting misinformation because they reproduce an outdated view or a misconception of the issue. In conclusion, the brief study shows that checking the sources for environment and climate information is more important than ever. That is the only way to ensure the information is correct. Yet recognizing wrong answers often requires detailed specialist knowledge of the topic in question, especially since the responses look plausible at first sight. The researchers also point out that, before every case of proposed use, consideration should be given to the energy consumption of language models and to the emissions associated with training the models.
Publication
Jens Bergener, Maike Gossen, Marja Lena Hoffmann, Felix Bießmann, Marek Veneny, Ruben Korenke
Evaluating the Quality of ChatGPT’s Climate-related Responses https://www.oekologisches-wirtschaften.de/index.php/oew/article/view/1978, Ökologisches Wirtschaften 2023
Further Information:
On the research project Green Consumption Assistant https://green-consumption-assistant.de/en/home-english/
Further information is available from:
Dr. Maike Gossen
TU Berlin
Chair of Social-ecological Transformation and Sustainable Digitalization
Email: maike.gossen@tu-berlin.de
Phone: 030 314-78868
Criteria of this press release:
Journalists
Environment / ecology, Information technology
transregional, national
Research projects, Research results
English
You can combine search terms with and, or and/or not, e.g. Philo not logy.
You can use brackets to separate combinations from each other, e.g. (Philo not logy) or (Psycho and logy).
Coherent groups of words will be located as complete phrases if you put them into quotation marks, e.g. “Federal Republic of Germany”.
You can also use the advanced search without entering search terms. It will then follow the criteria you have selected (e.g. country or subject area).
If you have not selected any criteria in a given category, the entire category will be searched (e.g. all subject areas or all countries).
