For those of you not of a certain age, at the climax of the original Ghostbusters movie, Gozer the Gozerian (an ancient Sumerian demigod, aka Google) offers the Ghostbusters a choice (albeit, not a great one): they get to choose the form of the monster that will annihilate humanity. As Hollywood movies often do, this worked out okay; evil marshmallow monsters can end up toasted. Even though Google isn't a giant Stay-Puft Marshmallow Man, the choice it's offering publishers is absolutely existential.
To provide search results pages and AI Overviews, Google crawls the web with its Googlebot. The company states that AI Overviews are part of its commitment to serve higher quality information and create opportunities for publishers. However, publishers face a dilemma: they can't block Google's AI from summarizing their content without also hampering their site's discoverability in search results, as the same Googlebot serves both purposes.
While Google allows publishers to block specific pages (or parts of pages) from appearing in AI Overviews, doing so would likely prevent those snippets from appearing across all of Google's search features, including web link listings. This puts publishers in an impossible position, as many rely heavily on search engines for traffic.
As you can imagine, the situation is a cause for concern among publishers. There are many times where Alphabet and Google get unfairly accused of monopolistic behavior, but this isn't one of them. We need a solution for Google's newly invented Publishers Dilemma. Hey Google… Can you give us an alternative to annihilation by deindexing or annihilation by scraping? I'm thinking of compensation for content creators and publishers, but I'll leave it to you to make a better offer.
As always your thoughts and comments are both welcome and encouraged. Just reply to this email. -s
P.S. If you want to get a better understanding of how AI bots scrape the web for model training, consider taking our free online course, . It will help you jumpstart your AI journey.
ABOUT SHELLY PALMER
Shelly Palmer is the Professor of Advanced Media in Residence at Syracuse University’s S.I. Newhouse School of Public Communications and CEO of The Palmer Group, a consulting practice that helps Fortune 500 companies with technology, media and marketing. Named he covers tech and business for , is a regular commentator on CNN and writes a popular . He's a , and the creator of the popular, free online course, . Follow or visit .
