Indonesia temporarily blocked Elon Musk's Grok chatbot due to the risk of AI-generated pornographic content, becoming the first country to deny access to the AI tool.
The move comes after governments and regulators from Europe to Asia have condemned and some have opened inquiries into sexualised content on the app.
xAI, the startup behind Grok, said it was restricting image generation and editing to paying subscribers as it tried to fix safeguard lapses that had allowed sexualised outputs including depictions of scantily clad children.
The government views the practice of non-consensual sexual deepfakes as a serious violation of human rights, dignity, and the security of citizens in the digital space," Communications and Digital Minister Meutya Hafid said in a statement.
The ministry has also summoned X officials to discuss the matter.
Mr Musk said on X that anyone using Grok to make illegal content would suffer the same consequences as if they had uploaded illegal content. xAI replied to Reuters' email seeking comment with what seemed to be an automated response: "Legacy Media Lies".
X did not immediately respond to a request for comment. Indonesia, with the world's biggest Muslim population, has strict rules that ban the sharing online of content deemed obscene.
The European Commission had ordered X to retain all documents relating to its AI chatbot for longer while the bloc ensures compliance with its rules after condemning it for producing sexualised images, a spokesperson said on Thursday.
Sweden joined the chorus of criticism saying the AI-generated images were unacceptable, after the country's deputy prime minister was targeted by a Grok user's prompt this week.
Swedish Prime Minister Ulf Kristersson described the images as "a kind of sexualised violence" and said: "It's distasteful, unacceptable, offensive."
xAI has put some restrictions on the image generation function of its Grok chatbot on X after widespread backlash. Users had been able to ask Grok directly on X to edit photos of people, including removing items of clothing and putting them in sexualized poses - often without their consent. Grok then published these images in replies on the social media platform.
Read MoreGrok’s bikini images open up new frontier for abuse of women in India
Dead chickens and molotov cocktails sent to critics of Indonesia’s flood response
China warns Elon Musk’s Starlink satellites pose ‘safety and security’ risk