By Peter Weinberger

For nearly three decades, the business model of the internet was built on a simple bargain.

Publishers created content. Search engines like Google copied and indexed it. In return, search engines sent readers back.

That arrangement was never perfect, and it never actually paid the Courier for a story. But it did send readers our way. A local newspaper like the Claremont Courier could report a story, publish it online, and trust that readers looking for information about Claremont would find us through Google, Safari, Apple News, Facebook, Instagram, newsletters, email, or another digital path.

That model is now changing quickly. Faster than I expected. Faster than almost every publisher expected.

Website analytics show a sharp drop in suspected bot/crawler traffic from China and Singapore after blocking measures were added, while a new spike in traffic from Japan appears to be another automated crawler and is being reviewed. Notice there are very few from the U.S.

At the Courier, we have seen a sharp drop in referrals from Google and Safari in just the last few months. At the same time, our website is being hit by automated bots from around the world, including AI-related crawlers that scan and collect information for AI models at a scale human readers never could. These bots can make traffic look busy while doing little or nothing to support the journalism that created the information in the first place.

This is not just a Courier issue. It is happening across millions of websites. The reason is artificial intelligence.

Accessing information is changing — fast

More readers are no longer searching the web the way they used to. Instead of typing a question into Google and clicking through a list of links, they are asking AI tools for direct answers, using information AI has ingested by scanning websites like the Courier.

For users, this can feel convenient. For publishers, it changes everything.

The old search model sent readers to original sources. The new AI model often summarizes those sources. A reader may get the answer, but the newspaper, website, photographer, reporter, editor, or public agency that created the original information may get no visit, no subscription, no donation, no ad view, and no direct relationship with that reader.

That is a serious problem because original reporting is not free to produce. You may have heard me mention that in these pages once or twice.

AI does not replace fact-based stories. It depends on them. And it is the same fight we have had with Google over use of Courier content. They are not paying us. They are using us to provide information they cannot create. AI takes that one step further.

That is what makes this moment so important. The web is moving from a referral economy to an extraction economy. In the old model, publishers received traffic in exchange for access. In the new model, AI systems may collect, summarize and repackage information while sending far less value back to the people who created it.

If we block every AI crawler, we may make it harder for readers to find accurate Courier reporting when they ask questions about Claremont. If we allow everything, our work may be used without compensation or proper credit. If we block the wrong bots, we may hurt our visibility in search. If we allow the wrong bots, we may overload our site and distort our analytics.

We will fight back

So the Courier has to be careful. But we are also going to push back.

We need to better understand which traffic is human, which traffic is useful, which traffic is automated, and which bots are simply taking without giving anything back. We need better tools to separate real readers from machines. We need to block abusive crawlers while keeping our reporting visible to people looking for trustworthy local information.

Our web developer has already helped make this a reality, but more work must be done. Currently, we have bots from 12 countries trying to gain website access every day.

But the larger solution is not technical. It is about relationships. And that means the Courier has to work harder and smarter to build even more direct connections with readers.

We need readers to come directly to us. Bookmark our website. Sign up for our newsletter.

Become a subscriber to print and/or online. Share Courier stories with neighbors. Send us news tips. Support the nonprofit that supports local journalism in Claremont. Hundreds of people already do this, but we need more help.

The Courier will focus on promoting more individual stories to bypass AI. We are adding 10 social media posts per week, using a form of geofencing around the website, deploying the latest Cloudflare technology to push back bad bots, and using our Google Ads grant to promote specific stories beyond Claremont’s boundaries.

And that is just the beginning.

Nearly bot proof

The Courier has something many local news publishers do not have: a trusted newspaper.

Unlike many digital-only publications, we are still supported by a print edition with deep community trust. The print edition remains central to our mission and our business. It is not directly affected by AI crawlers. It is far harder for AI systems to scrape at scale. And it arrives as a finished, edited, fact-checked product, produced by people who live in and care about this community.

In an era of misinformation, that matters. And it is only becoming more important.

Print may seem old-fashioned to some, but it has qualities that are becoming more valuable, not less. It is finite. It is polished. It is accountable. It has a beginning and an end. It reflects judgment. It tells readers: this is what we believe mattered this week in Claremont.

That does not mean we are turning away from digital news. Far from it. The Courier will continue publishing online. But we will be honest with our readers about what is happening. The online business model is changing fast, and local journalism cannot survive by pretending nothing has happened.

AI will become part of how people find information. Some of that will be helpful. Some of it will be harmful. The challenge is making sure original, fact-based reporting is not weakened in the process. And that is not going to happen at the Courier on my watch.

For more than a century, the Courier has adapted to new technology: hot type, offset printing, photography, desktop publishing, the internet, social media, smartphones, newsletters, video and now artificial intelligence. The tools and technology keep changing. The mission does not.

That is exactly why local journalism matters more than ever.

How you can help

• Got a news tip? Contact us at editor@claremont-courier.com.
• Read us online? Bookmark our website at: claremont-courier.com
• Support our mission by subscribing at: claremont.circ.news/about
• Donations are critical for salaries and printing expenses: claremont-courier.com/donations