AI search · Technical SEO
Why your site thrives on Google but is invisible to AI
Your site can rank well on Google, get green marks in SEO tools, and still be almost invisible to AI search and AI agents. ChatGPT, Claude, Perplexity and Gemini do not see the web the way Google does. In this article we go through three reasons why AI cannot see your page, how to test it yourself, and why server-generated HTML is what solves the problem.
In short: if your important content (headings, text, prices, Schema data) is loaded only in the browser with JavaScript, many AI bots see an empty page. The same applies when a firewall or CDN blocks the AI bot. Your SEO tool may not tell you anything about this.
Google and AI search read the web in different ways. Googlebot renders JavaScript and usually gets through firewalls. Most AI bots do not run JavaScript; they read only the raw HTML the server returns, and they can be blocked at the firewall or CDN.
That is why a site can thrive on Google but remain invisible to AI. Ordinary SEO tools do not notice this, because they read mainly the robots.txt file rather than the content actually served to bots.
The solution is for the critical content and Schema data to be ready in the HTML the server returns (SSR or a static page) and for AI bots not to be blocked. Novavalo produces exactly this kind of static HTML, so sites built with Novavalo are visible both to Google and to AI.
Different bots for different purposes
AI companies do not have a single bot but several separate bots for different tasks: one gathers material to train the model, another builds a search index, and a third fetches the page at the moment a user asks for it. Your site can allow one and block another without realising it.
For example, OpenAI and Anthropic each run more than one bot. At OpenAI, GPTBot gathers training data, OAI-SearchBot builds search results and ChatGPT-User fetches a page from a user's real-time request. At Anthropic, likewise, ClaudeBot gathers material, Claude-SearchBot checks pages for search and Claude-User retrieves a page when someone asks Claude something about that address.
Google is the exception here. It does not have a separate AI bot in the same way; instead, the traditional Googlebot largely handles the background work for AI Overview results and AI features too. That is why what works for Google does not automatically work for other AI providers: Google is already inside, whereas separate AI bots have to be let in on their own and they need to be able to read the page.
Firewalls and CDNs block bots before robots.txt
A protection layer, such as Cloudflare, a WAF firewall or your web host's blocking settings, can turn away an AI bot at the IP level, before the bot even gets to read the robots.txt file. In that case the bot does not get the page at all, no matter how good your content is.
Many CDN and security services have started to block known AI bots by default or with a single click. That can be a deliberate choice, but often it has been left on by accident or set up by someone else. The result is that a human visitor and Googlebot can reach the page, but ClaudeBot, for example, gets a block in response.
In a real-world example, ChatGPT could reach a site, but Anthropic's Claude was blocked by the server's firewall. To the user this shows up as one AI assistant being able to tell them about a company while another finds nothing about it. An ordinary SEO tool does not reveal the problem, because it reads only the robots.txt file, where the block does not appear: the block happens at the network level much earlier.
Robots.txt does not always hold
Another surprising point is that robots.txt is not binding for all bots. Bots acting on a user's request often ignore the rules. OpenAI has itself stated that because a ChatGPT-User request is triggered by a human, robots.txt rules do not necessarily apply to it.
This is not just theory. According to industry studies, ChatGPT-User reached pages on about half of the European sites that had explicitly blocked it in robots.txt. The practical lesson runs both ways: you cannot rely on robots.txt to stop real-time agents, and conversely you cannot assume that allowing a bot in robots.txt alone is enough to make your site visible to AI. Visibility is settled only by whether the bot can reach the page and whether it can read the content.
JavaScript and rendering: the most important technical point
Googlebot can run and render JavaScript, but most AI bots do not execute JavaScript at all. They read only the HTML the server returns. If important content is brought in only in the browser with JavaScript, the AI bot sees what is effectively an empty page.
This is the most important technical finding of the article. According to a broad study by Vercel and MERJ (December 2024), GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot did not run JavaScript. They may download the JavaScript files (as part of the data they fetch), but they do not execute them, so content added later by code stays hidden from them. Googlebot and some browser-based agents are exceptions that do render the page.
In practice the problem arises when a page is built with a heavy client-side framework, or when prices, product details, text or Schema data are injected onto the page only through Google Tag Manager or another script. To a human and to Google the page looks complete, but to the AI bot the body returned by the server is almost empty. The solution is for the content to be ready in the HTML the server generates, either server-rendered (SSR) or as a static page.
The page body is an almost empty <div id="root"></div>, and the headings, text and prices appear only once the browser runs the JavaScript. The AI bot reads only the empty shell.
The H1 heading, body text, prices and application/ld+json are directly in the HTML the server returns. The bot sees the same content as a human, without JavaScript.
How do you test whether AI can see your page?
You do not need special tools. With three checks you can quickly see whether AI can see your page's content.
Three ways to test visibility to AI
- Check the page source. In your browser, open View Source, not the DOM view in DevTools. Look for whether the H1 heading, body text, prices and
application/ld+jsonare found directly in the HTML. If they are missing from the source, they are missing for the AI bot too. - Ask the models directly. Ask ChatGPT, Claude and Perplexity to fetch information in real time directly from your page address. See whether they get the content or not. If a model finds nothing, it has a block or a rendering problem.
- Look at the server and CDN logs. Check in the server logs and the CDN control panel (for example Cloudflare's AI view) whether AI bots receive a 200 OK or blocks such as 403 or 429 in response. A block code tells you the bot is being turned away at the network level.
Do you need llms.txt and new magic files?
New separate files, such as llms.txt, are not necessarily needed. The major AI providers have not committed to them, and no single file fixes the fact that a bot cannot reach the page or cannot see the content. About 80 percent of ordinary SEO still applies.
In other words, the basics still carry: technical accessibility, a clear page structure, sensible internal linking and credibility (E-E-A-T, that is experience, expertise, authoritativeness and trustworthiness). The biggest change in the AI era is that a page's critical content and Schema data must be readable directly from the HTML without JavaScript run by the browser, and that bots must be let onto the page. Once this is in order, separate magic files are at most fine-tuning.
| Factor | Googlebot | Most AI bots |
|---|---|---|
| Runs JavaScript | Yes, renders the page | No, reads only the server's HTML |
| Usually gets through the firewall | Usually yes | Often blocked at the IP or CDN level |
| Follows robots.txt | Yes | Varies, a user request often ignores it |
| Sees content loaded only with JS | Yes (after rendering) | Does not see it |
How Novavalo solves this
The problems described above arise largely from how a site is built technically. Novavalo's approach removes them from the outset, because the pages are produced ready-made on the server.
Pages that are visible both to Google and to AI
Novavalo produces static HTML built in advance on the server that AI bots can read:
- The content is ready in the HTML. Headings, text, prices and Schema data are already in the server's response, not hidden behind JavaScript.
- No needless bot blocks. The pages are served through a fast CDN so that AI bots get in just as Google does.
- Structured data automatically. Each page is given appropriate Schema markup that AI can read and cite.
- A GEO/AEO visibility test. You can check in plain language how AI search sees your business, and get suggestions for fixes.
Frequently asked questions
Why does my site show up on Google but not in AI search?
Googlebot renders JavaScript and firewalls often let it through, so Google sees the page. Most AI bots do not run JavaScript; they read only the raw HTML the server returns, and they can be blocked at the firewall or CDN before they ever read the page. If the content appears only in the browser, or the bot is blocked, the AI sees an empty page.
Do AI bots run JavaScript?
Mostly they do not. According to a broad study, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot do not run JavaScript; they read only the HTML the server returns. They may download the JS files, but they do not execute them. Googlebot is the exception, because it renders JavaScript.
Why does my SEO tool show green even though AI cannot see the page?
Ordinary SEO tools often check only the robots.txt file and indexability. They do not notice if a firewall or CDN blocks the AI bot at the IP level, nor that critical content is loaded only with JavaScript. That is why a tool can show everything as fine even though, for the AI bot, the page stays empty or blocked.
How do I test whether AI can see my page?
Open the page source (View Source) and check whether the H1 heading, body text, prices and the application/ld+json schema are found directly in the HTML. Then ask ChatGPT, Claude or Perplexity to fetch information directly from your page address and see whether it succeeds. Finally, check in the server or CDN logs whether AI bots receive a 200 OK or blocks such as 403 or 429 in response.
Do I need an llms.txt file?
Not necessarily. The major AI providers have not committed to llms.txt, and it does not replace the basics. About 80 percent of ordinary SEO still applies: accessibility, a clear structure, internal linking and credibility (E-E-A-T). The most important change is that critical content and Schema data must be readable directly from the HTML without JavaScript.
Are Novavalo pages visible to AI?
Yes. Novavalo produces static HTML that is built in advance on the server, where headings, text, prices and Schema data are included without a JavaScript dependency. The pages are served through a fast CDN with no IP blocks on AI bots, so the AI bot sees the same content as a human.
Be visible in AI search too
Let AI build your business a site whose content is ready in the server's HTML and that both Google and AI search can see. You will see the result in a few minutes.
Start for free →Transparency note: This article was produced with the help of AI and Novavalo's editorial team is responsible for its content. The article is general information about AI search and technical SEO. Bots and their practices change quickly, so check the up-to-date details in the official sources listed below.