{"id":13893,"date":"2024-06-24T09:18:13","date_gmt":"2024-06-24T01:18:13","guid":{"rendered":"https:\/\/www.1ai.net\/?p=13893"},"modified":"2024-06-24T09:18:13","modified_gmt":"2024-06-24T01:18:13","slug":"%e6%b2%a1%e6%9c%89%e6%8e%88%e6%9d%83%e4%b9%9f%e6%b2%a1%e5%85%b3%e7%b3%bb%ef%bc%8c%e5%a4%9a%e5%ae%b6-ai-%e5%85%ac%e5%8f%b8%e7%bb%95%e8%bf%87%e7%bd%91%e7%bb%9c%e6%a0%87%e5%87%86%e6%8a%93%e5%8f%96","status":"publish","type":"post","link":"https:\/\/www.1ai.net\/en\/13893.html","title":{"rendered":"It doesn\u2019t matter if there is no authorization. Several AI companies bypass network standards to crawl news publishers\u2019 website content"},"content":{"rendered":"

According to Reuters on Saturday, TollBit, a startup focusing on \"content licensing,\" recently announced to the press that it has been working on a \"content licensing\" program.publisher<\/a>issued a warning that several artificial intelligence companies arecircumvents<\/strong>Publishers use to block crawled contentCommon Network Standards<\/strong>and use the crawl forTraining Generative AI Systems<\/strong>.<\/p>\n

The news comes after AI search startup Perplexity<\/a> Issued against the backdrop of a public dispute between and media outlet Forbes over the same web standard. Currently, there is an ongoing dispute between tech and media companies overThe Value of Content in the Age of Generative AI<\/strong>A broader debate is taking place.<\/p>\n

Tollbit positions itself asdry AI Companies<\/a><\/strong>andPublishers willing to enter into major license agreements with them<\/strong>The \"matchmaker\".<\/p>\n

Forbes has accused Perplexity of being in an AI-generated summary of thePlagiarizing their stories<\/strong>However, the formerNot labeled<\/strong>sources, and without permission from Forbes.<\/p>\n

Also, Wired magazine published an investigative story last week and noted that Perpexity mayIt's bypassed.<\/strong>\uff08A \"Robots Exclusion Protocol\" (set by the news publisher) or other program that blocks web crawlers.<\/p>\n

\"7cf751a3-7c72-4fef-819a-33f405a244b4.jpg@s_2w_820h_547\"<\/p>\n

Image source: Pexels<\/p>\n

claim to bein the name of<\/strong>\u00a0More than 2,000 U.S. publishers<\/strong>The News Media Alliance, a trade organization of the U.S. Department of State, also expressed concern about this behavior - the \"no-crawl\" or \"no-capture\" mechanisms that AI companies have put in place for publishers.robots.txt<\/span>\"Tools such as this one fall on deaf ears. If AI companies can't stop mass crawling,\" said Danielle Coffey, president of the organizationFailure to pass<\/strong>Profit from valuable content, and no way for journalists toPayment of compensation<\/strong>. "<\/p>\n

Tollbit said that Perplexity is not the only violator of the \"no-crawl\" mechanism on publishers' websites. According to its analysis, \"a large number\" of AI platforms have bypassed this mechanism, which sets a \"no-crawl\" policy for AI platforms to crawl their own content.whitelisting<\/strong>\" - Indicates which parts of their site can be crawled.<\/p>\n

\"This means that AI platforms from multiple sources (not just one company) are choosing to bypass the\u00a0robots.txt<\/span>\u00a0protocol to retrieve content from the site,\" TollBit writes, \"and the more publisher logs we acquire, the more times this pattern appears.\"<\/p>\n

A number of publishers, including The New York Times, have already filed suit for these infringementsSuing AI companies<\/strong>.. Other publishers have signed licensing agreements with AI companies, and AI companies are willing to pay for content, although the two sides often disagree on the value of the material. Many AI developers argue that they get content for freeNo laws have been violated.<\/strong>.<\/p>","protected":false},"excerpt":{"rendered":"

As Reuters reported on Saturday, TollBit, a startup focused on \"content licensing,\" recently warned news publishers that AI companies are circumventing common web standards used by publishers to prevent content from being crawled, and are using the crawled content to train generative AI systems. The news comes in the context of a public dispute between AI search startup Perplexity and media outlet Forbes over the same web standards. There is currently a broader debate between tech and media companies about the value of content in the era of generative AI. Tollbit has positioned itself as a \"matchmaker\" between content-starved AI companies and publishers willing to strike major licensing deals with them. Forbes has accused Pe<\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[146],"tags":[155,961,1386],"collection":[],"class_list":["post-13893","post","type-post","status-publish","format-standard","hentry","category-news","tag-ai","tag-perplexity","tag-1386"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/posts\/13893","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/comments?post=13893"}],"version-history":[{"count":0,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/posts\/13893\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/media?parent=13893"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/categories?post=13893"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/tags?post=13893"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/collection?post=13893"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}