---

---

15.07.2026 08:34

Dataset Filtering Provides Only Limited Protection Against CSAM Generation in Text-to-Image Models

Felix Koltermann Unternehmenskommunikation
CISPA Helmholtz Center for Information Security

Text-to-image models have made it significantly easier to generate synthetic images, including harmful and illegal content such as Child Sexual Abuse Material (CSAM). In public debates and policy discussions, a commonly proposed safeguard is to remove images of children from training datasets. However, a study led by CISPA-Faculty Dr. Ana-Maria Cretu suggests that this approach provides only limited protection and can be circumvented. She presented the findings of the study titled “Evaluating Concept Filtering Defenses against Child Sexual Abuse Material Generation by Text-to-Image Models” at the 47th IEEE Symposium on Security and Privacy (S&P 2026).

“Even before the rise of generative AI, there was already a serious problem of scale with Child Sexual Abuse Material (CSAM): Increasing amounts of CSAM were being shared online, and policymakers were looking for ways to limit its spread,” explains Ana-Maria Cretu. The emergence of generative AI has intensified these concerns because text-to-image models can be misused to create CSAM with unprecedented ease, harming children by damaging the reputation and mental health of victims depicted. Even when it does not depict a real child, AI-generated CSAM may hinder the investigation of real CSAM cases by wasting efforts to identify real children in content that looks real but is fully AI-generated.

“One proposal to design safer models that has gained attention is filtering images of children from training datasets,” says Cretu. “However, there was little evidence showing whether such approaches actually work. From both a policy and scientific perspective, it seemed important to evaluate these claims.” The study therefore set out to examine whether removing images of children from training data can effectively reduce a model’s ability to generate images of children.

Filtering Helps, but Does Not Solve the Problem

To answer this question, the researchers evaluated more than twenty automated methods for detecting and removing images containing children using their own evaluation framework. “We wanted to understand how effectively images of children can be identified and removed from large-scale training datasets,” says Cretu. The researchers found that the evaluated methods detected approximately 94 percent of images containing children. “While that may sound high, it is far from perfect,” Cretu explains. “In datasets containing billions of images, even a small error rate leaves a substantial number of child images behind. Our conclusion is that current child detection methods are not accurate enough to comprehensively filter children from large training datasets.” The most accurate methods were also computationally expensive.

Testing the Limits of Dataset Filtering

In a second phase, Cretu and her collaborators investigated whether filtering these images actually reduces a model’s ability to generate images that depict children. They trained text-to-image models from scratch on both filtered and unfiltered datasets using two publicly available image-caption datasets. They compared how difficult it was to generate a benign proxy concept for CSAM: children wearing glasses. Their results showed that filtering does make such generation more difficult, but only to a limited extent, requiring up to 10 additional queries to obtain the desired result. “Filtering makes images that depict children harder to generate, but not hard enough to stop a determined user,” Cretu explains. “Someone intent on misusing these systems would still be able to achieve their goals with relatively modest additional effort.”

Collateral Effects on Model Behavior

The study also revealed potential unintended consequences of dataset filtering. Images containing children often include many other concepts, such as parents, toys, and playgrounds. “When these images are removed from training data, the frequency of all related concepts is reduced as well,” Cretu explains. “This can affect a model’s broader capabilities and introduce unintended biases.” To investigate these effects, the researchers tested prompts such as “mother.” One notable finding was that models trained on unfiltered datasets frequently generated images depicting mothers together with babies. In filtered models, the babies disappeared—as expected—but the women also appeared noticeably older. These findings suggest that filtering can alter model behavior beyond its intended objective.

Outlook

Cretu emphasizes the need for greater transparency and more rigorous evaluations of safety measures: “We believe companies developing models with image-generation capabilities should evaluate their safeguards transparently, test them against realistic adversaries, share the results of AI safety testing, and assess potential collateral effects.” She notes that many companies publicly support implementing child safety recommendations without being transparent about which measures have actually been deployed or how effective they are. For researchers, many questions remain open: “We need better methods for evaluating capabilities in AI models and, in the case of illegal capabilities like CSAM generation, we need better proxies that model adversarial intents while allowing researchers to red-team models in an ethical and legal manner.” Ultimately, Cretu’s goal extends beyond assessing a single safety mechanism. Her aim is to establish a more rigorous and transparent process for evaluating AI safety interventions overall.

---

Originalpublikation:

Crețu, Ana-Maria; Kireev, Klim; Abdalla, Amro; Obinna, Wisdom; Meier, Raphael; Bargal, Sarah Adel; et al. (2026). Evaluating Concept Filtering Defenses against Child Sexual Abuse Material Generation by Text-to-Image Models. CISPA. Conference contribution. https://doi.org/10.60882/cispa.32771970.v1

---

<
Visualization to the paper “Evaluating Concept Filtering Defenses against Child Sexual Abuse Materia ...

Copyright: CISPA

---

Merkmale dieser Pressemitteilung:
Journalisten
Informationstechnik
überregional
Forschungsergebnisse
Englisch

---