Web crawler

Meta has quietly released a new web crawler that searches the internet and collects large amounts of data to power its artificial intelligence models. Launched last month, Meta's new web crawler, Meta External Agent, is similar to OpenAI's GPTBot and can scrape AI training data on the web, such as text in news articles or conversations in online discussion groups, according to three companies that track web crawlers. Meta did update a new web crawler for OpenAI in late July, according to usage archive history.

Cloudflare on Wednesday gave web hosting customers a way to block AI bots from crawling website content and using data to train machine learning models without permission. It was based on customer dislike of AI bots and said to help protect content creators on the internet, the company said in a statement. Image source note: Image generated by AI, image licensing service provider Midjourney "We heard loud and clear that customers don't want AI bots to access their websites, especially those that do so dishonestly. To help, we added a...