3 ways to optimize for AI search bots

Advances in AI search are opening a new chapter, with consumers exploring Google alternatives.

• ChatGPT has hit a new milestone of 300 million weekly active users.
• Perplexity has grown to serve 100 million queries per week.
• Google’s global market share trended below 90% for the first time since 2015.

The rapid rise of AI search platforms brings significant opportunities and growing challenges. 

On one hand, brands can boost visibility and drive demand like never before. 

On the other hand, they face new hurdles, such as copyright concerns, increasing infrastructure costs, and the ongoing challenge of measuring ROI.

Despite shifts in search interfaces and consumer behavior, user intent has stayed the same.

People want to find information – whether it’s through a catalog, a search engine, or an AI platform.

Today’s tools simply help consumers get from point A to point B faster. 

This new efficiency challenges brands to rethink how their content is discovered and delivered – so they don’t get lost in an increasingly complex search landscape.

The new bot landscape

For two decades, search has demanded time and effort from consumers. 

Now, AI search simplifies and centralizes the customer journey directly on an AI platform. 

As a result, we can expect traffic trends to shift as bots take on the bulk of discovering and sharing website content with consumers.

AI search platforms are taking over more tasks that consumers used to handle. 

As a result, predictions like Gartner’s – expecting a 25% decline in search engine volume by 2026 due to AI chatbots and virtual agents – are becoming more likely. 

This shift is driven by the rise in bot traffic and the decrease in human traffic.

But what exactly is the new bot landscape? 

Several kinds of crawler bots influence AI results. 

• Some bots, like OpenAI’s OAI-SearchBot, scrape and index the web like traditional search engines like Google and Bing, aiming to improve the relevance and accuracy of what users see. 
• Others, like OpenAI’s GPTbot, use web data to train and refine their large language models (LLMs). 
• Still, more (like OpenAI’s ChatGPT-User) tap into an existing search index, usually Bing’s, to provide live results.

All crawler bots use similar methods to discover and navigate websites, but AI-powered crawlers operate differently from traditional search engine crawlers.

Leveraging natural language processing (NLP) and machine learning, AI crawlers interpret content with a deeper understanding of context, intent, and nuance. 

Since AI models can only reference the data they know about, it’s essential to ensure AI crawlers find the most relevant content about your brand and products. 

As of February, ChatGPT’s core knowledge is based on data up to June 2024, resulting in a lag of over seven months. 

This means it can’t provide real-time information like the seven-day weather forecast or the latest shopping deals.

However, these platforms use retrieval-augmented generation (RAG), relying on real-time crawlers and indexes such as Bing’s to augment and provide real-time responses. 

If AI platforms aren’t aware of your brand, they can’t reference it in generative conversations with consumers. 

Optimizing for these bots ensures your brand remains visible and competitive.

Dig deeper: AI optimization – How to optimize your content for AI search and agents

3 ways to start your bot optimization journey

1. Begin with an audit 

To optimize for bots, start by understanding their actions on your site and how the data is processed during indexing or training. 

Begin with a technical SEO audit, as the same challenges that historically affected Googlebot – like indexing issues – will also impact these newer, less sophisticated bots and AI engines.

Next, review how your content – and your competitors’ – is represented across different search and AI platforms. 

What opportunities and gaps do you see? 

Remember, if your content isn’t crawled, it won’t be indexed, used to train AI models, or seen by consumers. 

This step helps you decide what content to expose, consolidate, or block from AI bots.

Consider analyzing your log files to:

• Understand how bots find your content.
• Identify their crawling patterns, scale, and velocity. 

Parse user agent logs to identify the bots visiting your site – like Bytespyder (TikTok), GPTBot (OpenAI), or ClaudeBot (Anthropic). 

What are they consuming and how much? 

Combine this with traffic data and analytics to find patterns between crawl and traffic, giving you a clearer picture of ROI, which will inform your governance plan.

Analyzing log files isn’t just technical – it’s strategic. 

By understanding bot behavior, you can identify performance issues, optimize site efficiency, and improve visibility in both traditional and AI search.

2. Determine your goals and develop a governance strategy that prioritizes ROI

Reflect on your website and traffic goals and how they align with the use of your content.

Analyze the cost breakdown, including:

• The opportunity cost of bots crawling your site.
• The impact on your infrastructure.

Once you understand your intended ROI, develop a governance plan with organizational buy-in to decide which bots should be allowed to crawl your site and which to block.

Notably, publishers are at the forefront of blocking bots to prevent content scraping, copyright issues, and content misuse.

Once you’ve identified the priority bots for your brand, update search engines to recrawl your content so it can be referenced in AI-generated results.

To do this:

• Keep your sitemaps updated.
• Ping protocols like IndexNow.
• Even submit content directly to Bing for indexing.

Dig deeper: 3 reasons not to block GPTBot from crawling your site

3. Optimize, refine, and don’t ignore the fundamentals

Just like traditional SEO, the new search landscape requires continuous optimization – it’s not a “set it and forget it” situation. 

We must keep refining our strategies and stick to proven best practices. 

Maintaining the fundamentals of technical SEO and site health is as important as ever. This includes:

• Strong information architecture.
• Up-to-date sitemaps.
• Addressing issues like thin or duplicate content.

Performing well in organic rankings remains one of the most influential factors. 

For example, in Google’s AI Overviews:

• Three-quarters of the links also rank in position 12 or higher in organic search.
• 90% of all AI Overview links come from positions 35 or higher. 

Since many AI platforms pull fresh content from organic search indexes, your rankings directly affect brand visibility in AI search. 

Even if consumers don’t click on those links, your organic rankings still impact brand discovery.

To stay visible, focus on your most valuable content, track what’s performing well, and identify areas for improvement. 

The bonus? 

Strong organic rankings help you in more than just AI Overviews. 

They also improve visibility in Google search, Meta AI, virtual assistants like Siri, and other AI platforms.

Dig deeper: 6 easy ways to adapt your SEO strategy for stronger AI visibility

The road ahead

There are no hard-and-fast rules – yet. 

We know that SEO fundamentals still matter, and we’re all learning what works as the search landscape evolves. 

Strategies will vary by industry – whether you’re a publisher like Search Engine Land or a retailer like Nike – but there’s plenty of opportunity ahead, even with the work still to be done.

Dig deeper: Your 2025 playbook for AI-powered cross-channel brand visibility