Meta’s latest AI improves its terrible content moderation, just a little

Meta has revealed it’s tested using AI for content moderation chores and found it does better than humans.

The social networking giant on Thursday announced it has started a global rollout for its Meta AI support, a tool that handles tasks like password resets, reporting dodgy content, explaining content takedowns and allowing appeals, or managing privacy settings.

The company also said “Over the next few years, we will deploy more advanced AI systems across our apps to transform our approach to content enforcement, more accurately finding and removing severe content violations like scams and illegal content, so people see less of them.”

Early experiments have delivered promising results: one AI tool detected and mitigated 5,000 attempts at scamming users to reveal their passwords every day. Meta says its human teams could not detect those scams.

Another AI helped to reduce the number of reports users lodged about fake celebrity profiles by over 80 percent. Other tests doubled detection of adult sexual solicitation content that violates Meta’s rules.

Meta says its AI can also “Prevent an account takeover by noticing it was suddenly accessed from a new location, the password was changed, and edits were made to the profile.” The company says those changes “look harmless to a person reviewing the account, but AI was able to recognize as a threat.”

• Chatbot Romeos keep users talking longer, but harm their mental health

• Meta reveals four Broadcom-built custom AI chips, claims some outperform commercial silicon

• AI nonsense finds new home as Meta acquires Moltbook

• Facebook went down for about three hours, interrupting your poking and Meta's ads business

That’s an odd observation given that numerous enterprise security products can detect “impossible travel” such as a single user logging in from London and an hour later requesting a password reset from San Francisco, and flag it as a likely attack.

Meta also enthused that AI can “Detect a fake site spoofing a legitimate web address and pretending to be a popular sporting goods store by noticing the real logo being used with unusually low prices and a suspicious web address,” because AI “drove down views of ads with scams and other serious violations by seven percent, offering promising results and better protections for users and brands.”

Again, that’s a nice outcome, but also a little odd as fake ads are a known problem – and one to which Meta has often been indifferent.

Your correspondent once spotted a suspicious ad for a brand that publishes lists of its legitimate URLs. The ad led to a spoof site, so I made a report to Meta – which replied that the fake site was not in violation of its policies. The company ignored my reply that pointed out the URL was not on the brand’s list of official sites.

Leaving the metaverse

Meta’s march toward AI came in the same week as it walked away from the metaverse, its vision for immersive online communities.

On Wednesday, the company announced the shutdown of Horizon Worlds, its metaverse platform.

That decision meant owners of Meta’s Quest VR goggles would have a lot less content to consume. The company later walked that back and promised to continue offering some of its immersive environments but said it would not create new ones.

Meta adopted its current name to reflect founder Mark Zuckerberg’s belief that the metaverse was the next big thing, and spent over $80 billion on its ambitions over five years – a period when we can now see its content moderation for Facebook and Instagram was going badly, and children were often harmed by the company’s products.

The company now plans to develop something it calls “superintelligence,” and is spending tens of billions more to make that happen. ®