In the age of AI and big tech bullying, online publishers are suddenly caught between a rock and a hard place. They now have to let Google use their content for AI-generated search answers, or risk losing their visibility in search results.
How to Use AI for Web Scraping with Excel and Google Sheets
Google's web crawler, Googlebot, scans websites to generate search results and is the worm at the root of this problem, as the worm now serves a dual purpose: collecting data for both traditional search results and AI-generated answers.
This means that publishers who block Google from using their content for AI answers also prevent their pages from appearing in standard search results. Since visibility in search results is crucial to their survival, this puts them in a difficult position.
Kyle Wiens, CEO of iFixit, highlighted this challenge, stating that blocking Google's crawler would result in lost traffic and customers.
