AI Deepfake Regulation and Grok Crisis: Why Banning Is Not The Answer

A comprehensive analysis of AI-generated non-consensual imagery, platform accountability, and why banning is not the answer, fixing is

By Aarti Samani | Social Engineering, Deepfake Fraud & AI Safety Expert

Calls to ban Grok are understandable. When LBC asked me whether the platform should be banned in the UK, I argued for something different: don’t ban it, block it globally until xAI fixes it. The debate around AI deepfake regulation is about holding platforms to the same high safety standards we apply to cars, food, and medicines.

🎙️ Listen to my LBC interview

What happened with Grok and deepfakes?

In late December 2025, users discovered they could tag Grok, xAI’s AI chatbot on X (formerly Twitter), and request image edits from posts. What began as requests to place people in bikinis rapidly escalated into what Reuters termed a “mass digital undressing spree.”

Women’s photos were digitally stripped without consent. Then, horrifyingly, the same began happening to images of children. Grok itself acknowledged generating sexualised images of girls it estimated to be aged 12–16.

Key findings from researchers:

Copyleaks, an AI detection platform, found that Grok was generating roughly one non-consensual explicit image per minute. In a single week, they identified thousands of explicit Grok-generated images.

Researchers at AI Forensics, a European non-profit, analysed over 20,000 random images generated by Grok between 25 December 2025 and 1 January 2026. More than half of the images depicting people,53%, showed individuals in minimal attire such as underwear or bikinis. Of those, 81% were individuals presenting as women.

This was happening on one of the world’s largest social media platforms, using a free tool available to everyone. This was not a dark web problem.

Should Grok be banned in the UK?

Grok should be fixed. Until adequate safety guardrails are implemented, it should be blocked. Not just in the UK, but globally.

This is an established practice, it’s not radical. It follows the same principle applied to consumer products in other sectors.

  • Automotive: We recall unsafe vehicles
  • Food: We pull contaminated food from shelves
  • Pharmaceuticals: We do not allow medicines into the market until clinical trials prove they are safe.

AI platforms that predictably cause harm should meet the same high standards, as other consumer products. They should be restricted until safety requirements are met.

The distinction matters: a ban suggests the technology itself is the problem. A block pending safety compliance places responsibility where it belongs, on the company executives who chose to deploy an unsafe product.

Why did Grok generate harmful content when other AI tools don’t?

The technology to prevent non-consensual explicit imagery exists. Other companies use it. xAI chose not to.

Steven Adler, a former OpenAI safety researcher, told CNN: “You can absolutely build guardrails that scan an image for whether there is a child in it and make the AI then behave more cautiously. But the guardrails have costs.”

Evidence this was a deliberate choice, not a technical limitation:

  1. Leadership pressure against safety measures: CNN reported that at a meeting in recent weeks before the controversy erupted, Elon Musk was “really unhappy” about restrictions on Grok’s image generator. Sources told CNN that Musk has “been unhappy about over-censoring” on Grok “for a long time.”
  2. Safety team departures: Three members of xAI’s already small safety team publicly announced their departures in the weeks preceding the crisis, including the head of product safety.
  3. Dismissive response to concerns: When journalists contacted xAI for comment, the company’s auto-reply read “Legacy Media Lies.” When users flagged child safety issues, Musk reportedly responded with laugh-cry emojis.

This pattern does not reflect a company that made an honest mistake and is working urgently to correct it.

Is regulating AI deepfakes a free speech issue?

No. This is a consent and safety issue, it is not a free speech issue.

The question is not whether AI should generate images. The question is very specific:

Should platforms be required to prevent non-consensual explicit images
of identifiable, real people, including children?

Yes. Absolutely.

Your right to expression does not extend to digitally undressing someone without their permission. The free speech framing is a deflection from the actual issue: companies profiting from tools that predictably violate individuals’ dignity and safety.

What are the current regulations on AI-generated non-consensual imagery?

International responses to the Grok crisis

AI deepfake regulation is accelerating globally. Indonesia became the first country to block Grok on Saturday, with Communication and Digital Affairs Minister Meutya Hafid stating that “the practice of non-consensual deepfakes” is “a serious violation of human rights, dignity, and the security of citizens in the digital space.”

Malaysia followed on Sunday, citing “repeated misuse” of Grok to generate obscene and non-consensual imagery.

In Europe, France has flagged the content as “clearly illegal” under the EU Digital Services Act, with the Paris prosecutor’s office opening an investigation.

The UK’s Ofcom has launched a formal investigation into whether X and xAI violated regulations meant to protect UK users, warning that X could face a ban or multimillion-pound fine.

For a comprehensive tracker of regulatory responses worldwide, Tech Policy Press is maintaining an updated list.

US legislation: The TAKE IT DOWN Act (2025)

The TAKE IT DOWN Act, signed into law by President Trump on 19 May 2025, makes it illegal to share online non-consensual intimate images, real or AI-generated, and requires platforms to remove such images within 48 hours of notification. However, the platform compliance provisions don’t take effect until May 2026.

Current enforcement gaps:

Individual victims have limited recourse in the meantime. Section 230 generally protects platforms from liability for user-generated content. And with Musk’s political proximity to the Trump administration, federal enforcement appears unlikely in the near term.

What should AI deepfake regulation and accountability look like?

Meaningful AI deepfake regulation requires three structural changes:

1. Pre-deployment safety requirements

AI image generation tools should pass child safety checks before public release, not after harm has occurred.

Current practice allows companies to treat users as unwitting beta testers for dangerous products.

2. Platform liability for foreseeable harm

When product design makes misuse predictable, the company shares responsibility for that misuse. xAI didn’t just host harmful content, its tool generated it. This distinction should carry legal weight.

3. Executive accountability

Directors and executives who authorise deployment of products without adequate safeguards must face personal consequences. Currently, they face none.

We don’t permit pharmaceutical executives to release untested drugs and walk away when patients are harmed. AI executives should not enjoy immunity for their negligence or deliberate decisions.

How can companies test whether their AI tools are safe?

The “Child’s School Photo” Test

A simple litmus test for any AI company or executive making deployment decisions:

If a stranger could use your tool to digitally undress a child’s school photo with a single prompt, you have a problem.

This test cuts through technical complexity and corporate justifications. There is no ambiguity. If your product fails it, fix it before releasing it to market.

What happens next for AI platform regulation?

The future of AI deepfake regulation is being written now. Companies treating safety as an afterthought will increasingly find themselves blocked from markets, subject to significant fines, and excluded from mainstream commercial deployment.

International pressure is mounting faster than domestic US enforcement. For AI companies operating globally, the question is not whether safety requirements will be imposed. It is when they will be imposed and at what cost to companies that failed to act proactively.

Key takeaways

AI deepfake regulation is accelerating. Companies without guardrails face market exclusion.

  1. The crisis was preventable: Technology to prevent non-consensual AI imagery exists and is used by responsible companies.
  2. This was a choice, not a bug: Evidence suggests xAI deliberately minimised safety measures despite internal warnings.
  3. Free speech framing is a deflection: The issue is consent and child safety, not censorship.
  4. Regulation is coming: International action is accelerating; companies without guardrails face market exclusion.
  5. Executive accountability is missing: Until leaders face personal consequences for deploying unsafe AI, the incentive structure rewards recklessness.

About the author

Aarti Samani is the founder of Shreem Growth Partners, specialising in deepfake fraud resilience and cybersecurity awareness for organisations. She is a former Chief Product and Marketing Officer at iProov, a biometric identity verification company, where she helped scale the business from ÂŁ1M to ÂŁ22M revenue. Aarti teaches executive education on AI safety and social engineering at Durham Business School, INSEAD, and MITxPro, and appears regularly on BBC and CNBC as an expert on deepfake threats and online safety.

If this analysis was useful, please share it. This conversation needs to reach the executives making deployment decisions, and the boards meant to hold them accountable.

Frequently Asked Questions

Should Grok be banned in the UK?

No, it should be fixed. Until adequate safety guardrails are implemented, it should be blocked globally. This follows the same principle we apply to other consumer products: we recall unsafe cars and pull contaminated food from shelves.

Is regulating AI deepfakes a free speech issue?

No. This is a consent and safety issue. The question is whether platforms should prevent non-consensual explicit images of identifiable real people, including children. Your right to expression does not extend to digitally undressing someone without their permission.

What is the “Child’s School Photo” Test?

A simple AI safety test: if a stranger could use your tool to digitally undress a child’s school photo with a single prompt, your product is not ready for public deployment. Fix it before releasing to market.

Why did Grok generate harmful content when other AI tools don’t?

The technology to prevent non-consensual imagery exists—other companies use it. Evidence suggests xAI deliberately minimised safety measures: internal pressure against guardrails, safety team departures, and dismissive responses to concerns about child safety.

What should AI platform accountability look like?

Three things: pre-deployment safety requirements before public release, platform liability when product design makes misuse predictable, and personal consequences for executives who authorise deployment without adequate safeguards.

Sources