Crawler Robot

Meta has quietly released a new web crawler that searches the internet and collects large amounts of data to power its artificial intelligence models. Launched last month, Meta's new web crawler, Meta External Agent, is similar to OpenAI's GPTBot and can scrape AI training data on the web, such as text in news articles or conversations in online discussion groups, according to three companies that track web crawlers. Meta did update a new web crawler for OpenAI in late July, according to usage archive history.