A report by AI Forensics has uncovered a significant issue with Hugging Face, an open-source AI platform valued in the billions, regarding nonconsensual deepfakes. Researchers tested nine of the top image editing Spaces on the platform and found that seven could easily transform a clothed image of a woman into a topless one. The findings highlight the platform's role in facilitating harmful content creation.
Further testing by AI Forensics involved setting up honey-pot-style image editing Spaces to track user interactions. Over a week, they received more than 1,000 prompts, with 73% being sexual in nature. Of these, 83% sought to undress or sexualize the person in the submitted image, and 95% of these were women. The research also noted that 6.7% of the sexual requests targeted apparent children. "Most of the Spaces tested can be used for generating nonconsensual intimate images, and users are actually using it for these purposes," said Paul Bouchaud, a lead researcher at AI Forensics.
Hugging Face has content policies prohibiting child sexual abuse material and nonconsensual deepfakes, but the company did not respond to WIRED's questions about its content moderation mechanisms. Some pages promoting nudifying services were removed after WIRED contacted the company, but it is unclear if this was related to the findings. The report underscores the broader issue of generative AI models being used to create harmful content, with many models trained on sexual images from the internet, enabling the creation of explicit content without safety mechanisms.
Source: wired
