According to October 29, since Google last year launched its diseased "AI Overviews" function, there has been widespread public awareness that there is a huge difference between the results of an artificially intelligent search and the list of traditional links that the search engine has provided for decades. Today, a new study quantifys this differenceShow AI Search EngineThey tend to refer to less frequent visits, even traditional onesGoogle SearchThe site is difficult to find out from the top 100 results。
1AI notes that in a pre-print paper entitled "Characterizing Web Search in The Age of Generating AI", researchers at the University of Pountroll, Germany, and Max Planck Institute for Software Systems compared Google Traditional Search results to their AI Overviews, Gemini 2.5-Flash and incorporated the GPT-4o web search model and the GPT-4o combination search tool (i.e., web search only when large language models determine the need for external information)。
Researchers select test search statements from multiple sources, including specific questions from the “WildChat” data central users to ChatGPT, political issues listed on the AllSides platform, and the 100 most frequently searched commodities on the Amazon。
The study found that:THE SOURCE OF INFORMATION CITED IN THE GENERATING AI SEARCH TOOL, WHOSE WEBSITE IS OFTEN LESS POPULAR THAN THE TRADITIONAL 10-PERSON SITE, based on measurements from the domain name tracking tool Transco. Compared to the links in the traditional Google search results, the websites cited in the AI engine are more likely not to be among the top 1,000 or the top 1,000,000 domain names in Transco. Of these, Gemini ' s search is particularly evident: the median ranking of its source falls above 1000 in Tranco, indicating a high reliance on unusual web resources。
IN ADDITION, THE REFERENCES TO THE AI SEARCH ENGINE ARE OFTEN ALMOST ABSENT FROM TRADITIONAL GOOGLE SEARCH RESULTS FOR THE SAME KEYWORD. FOR EXAMPLE, OF THE SOURCES CITED IN THE GOOGLE AI OVERVIEW, 53% DID NOT APPEAR IN THE TOP 10 RESULTS OF THE TRADITIONAL GOOGLE SEARCH FOR CORRESPONDING QUERIES; 40% SOURCES DID NOT EVEN ENTER THE TOP 100。
certainly,THESE DIFFERENCES DO NOT MEAN THAT AI PRODUCES RESULTS THAT ARE NECESSARILY WORSEI DON'T KNOW. THE STUDY FOUND THAT GPT-BASED SEARCHES TEND TO REFER MORE TO CORPORATE NETWORKS AND ENCYCLOPEDIA SITES AS SOURCES OF INFORMATION THAN TO SOCIAL MEDIA CONTENT。
AN ANALYTICAL TOOL BASED ON A LARGE-LANGUAGE MODEL SHOWS THAT THE NUMBER OF IDENTIFIABLE “CONCEPTS” COVERED BY AI SEARCH RESULTS IS COMPARABLE TO THE PREVIOUS 10 RESULTS OF THE TRADITIONAL SEARCH, INDICATING A SIMILAR LEVEL OF INFORMATION DETAIL, DIVERSITY AND NOVELTY. AT THE SAME TIME, HOWEVER, THE RESEARCHERS HAVE POINTED OUT THAT “GENERIC ENGINES TEND TO COMPRESS INFORMATION AND SOMETIMES OMIT SECONDARY OR VAGUE ELEMENTS THAT ARE RETAINED IN TRADITIONAL SEARCH RESULTS”. THIS IS PARTICULARLY TRUE IN THE FACE OF VAGUE SEARCH TERMS (SUCH AS NAMES SHARED BY MANY PUBLIC FIGURES), WHEN “INFORMATION ON TRADITIONAL SEARCH RESULTS IS MORE COMPREHENSIVE”。
On the other hand, AI search engines have the advantage of combining pre-trained “in-house knowledge” with references to information in web pages. This is particularly evident in the GPT-4o combination of search tools: The model often does not provide any external reference links, but rather provides answers based directly on its own training data。
However, this reliance on pre-trained data may also be short-set when searching for time-sensitive information. When researchers tested with the key words in the "real time heat search list" on September 15th, they found that "GPT-4o with search tools" often responded to questions like "Can you provide more details?" That kind of tip, instead of actively searching for up-to-date web information。
WHILE THE RESEARCHERS DID NOT ULTIMATELY DETERMINE THAT THE AI SEARCH ENGINE AS A WHOLE WAS BETTER OR WORSE THAN THE TRADITIONAL SEARCH LINKS, THEY CALLED FOR AN ENHANCED METHODOLOGICAL STUDY OF THE ASSESSMENT OF THE GENERATED SEARCH SYSTEM IN THE FUTURE, EMPHASIZING THE NEED TO TAKE INTO ACCOUNT THE DIVERSITY OF SOURCES OF INFORMATION, THE INTEGRITY OF THE CONCEPTUAL COVERAGE AND THE DIMENSIONS OF INFORMATION INTEGRATION CAPABILITIES IN ORDER TO BUILD A MORE SCIENTIFIC EVALUATION SYSTEM。