AI crawlers read your website best when the important content is already in the HTML your server sends back. Google runs JavaScript before it indexes a page. A measurement by Vercel and MERJ from December 2024 found that the crawlers of OpenAI, Anthropic and Perplexity did not. Text that only appears after JavaScript runs can therefore be missing for those AI crawlers.
What is the difference between server HTML and content added by JavaScript?
When a browser or crawler requests a page, your server sends back HTML. On many websites, the text is already in it. That is called server HTML, and when the server builds the page for each request, server-side rendering.
On other websites, the server sends an almost empty page with scripts. The browser runs those scripts and then fetches or builds the text itself. That is called client-side rendering. To a visitor it looks the same. To a crawler that does not run JavaScript, the page is then largely empty.
The same page, built in two ways
| Server HTML | Client-side rendering | |
|---|---|---|
| What the server sends | The full text, headings and links | An almost empty page with scripts |
| What a visitor sees | The whole page | The whole page, once the scripts have loaded |
| Crawler without JavaScript | The whole page | Mostly an empty page |
| Googlebot | The whole page, straight away | The whole page, after rendering |
Say an online garden furniture shop loads the FAQs on every product page through a separate app, with a script that only inserts the answers after the page has loaded. A visitor reads the answers as normal. A crawler that only reads the HTML sees a “Frequently asked questions” heading and nothing else. In the page source, that looks roughly like this:
<h2>Frequently asked questions</h2>
<div id="faq-app"></div>
<script src="/apps/faq-widget.js"></script>If the answers are in the server HTML instead, every crawler reads them:
<h2>Frequently asked questions</h2>
<h3>Can this lounge set stay outside all year?</h3>
<p>Yes, the frame is powder-coated aluminium. Store the cushions somewhere dry in winter.</p>How does Google handle JavaScript?
In Understand the JavaScript SEO basics, Google describes three stages: crawling, rendering and indexing. After fetching a page, Googlebot puts it in a queue for rendering. A current version of Chromium then runs the JavaScript, and Google indexes the page based on the rendered HTML. According to Google, a page can sit in that queue for a few seconds, sometimes longer.
How Googlebot processes a page with JavaScript
- CrawlingGooglebot fetches the HTML and checks whether robots.txt allows the URL.
- QueueThe page waits to be rendered. According to Google, usually a few seconds, sometimes longer.
- RenderingA current version of Chromium runs the JavaScript.
- IndexingGoogle uses the rendered HTML for the index and looks for new links in it.
So Google can read content from JavaScript, as long as it is not blocked. Even so, Google recommends server-side rendering or pre-rendering, because it makes the site faster for visitors and crawlers. Google explicitly adds that not every bot can run JavaScript.
What do we know about AI crawlers?
Here the information is thinner. An AI crawler is software from an AI company that fetches pages for training, search or a user’s request. In their crawler documentation, OpenAI, Anthropic and Perplexity describe what their crawlers are for, but not whether they run JavaScript (checked on 11 November 2026). The best public source is a measurement.
In December 2024, Vercel and MERJ published an analysis of crawler traffic on websites. Their findings:
| Crawler | Ran JavaScript in the measurement? |
|---|---|
| OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot) | No |
| Anthropic (ClaudeBot) | No |
| Perplexity (PerplexityBot) | No |
| Meta (Meta-ExternalAgent) | No |
| ByteDance (Bytespider) | No |
| Gemini, through Googlebot’s infrastructure | Yes |
| Applebot | Yes |
The crawlers of OpenAI and Anthropic did fetch JavaScript files, but did not run them. As a result, they could not read content that is built in the browser.
Bear the limits of that measurement in mind. It dates from December 2024, and crawlers may have changed since. Claude-SearchBot, Claude-User and Perplexity-User were not included. We have not found a more recent measurement that can be verified.
Microsoft does say something about it in its own documentation. The Bing Webmaster Guidelines ask you not to hide important content behind client-side rendering, because content that cannot be rendered reliably may not be indexed or chosen as a source. A post from the Bing team from October 2025 also advises against hiding important answers in tabs or accordions.
What belongs in the server HTML?
Everything a crawler needs to understand the page and use it as a source:
- The main text of the page.
- The page title, the meta description and the headings.
- Product information: name, price, availability, specifications.
- Answers to frequently asked questions. Read more in An FAQ that helps customers, Google and AI assistants.
- Internal links as ordinary
<a href>links, so crawlers can find your other pages. Microsoft mentions this in its guidelines too. - Structured data in JSON-LD.
Interactive parts such as a filter, a basket or a calculator can run on JavaScript. Vercel and MERJ gave the same advice: critical content on the server, secondary features in the browser.
What if your website runs on JavaScript?
You do not have to throw away a website that is built entirely in the browser. There are three common solutions, and your developer or builder will know which one suits your technology.
- Server-side rendering. The server builds the full HTML for every request. JavaScript in the browser then takes over the interactive parts.
- Pre-rendering or static generation. The HTML of each page is made in advance, for example every time you publish. That works well for pages that are the same for every visitor, such as articles and service pages.
- Moving individual parts. Often only one part runs on JavaScript, such as an FAQ app, a reviews block or a tab with specifications. Then it is enough to put that one part in the server HTML.
On Shopify and WooCommerce, the theme usually builds the page on the server. The risk there lies mainly in apps and plugins that insert text afterwards. So test again after every new app or plugin.
How do you check what a crawler sees?
- View the page source. Right-click the page and choose “View page source”. That is the HTML as the server sends it. Use Ctrl+F or Cmd+F to search for a sentence from your main text or an FAQ answer. Do not use “Inspect”: that shows the page after JavaScript has already run.
- Use curl. In a terminal,
curl -s https://yourdomain.com/page | grep "a sentence from your text"shows whether that sentence is in the server HTML. - Turn JavaScript off. Disable JavaScript in your browser settings and reload the page. Whatever you can still read, a crawler without JavaScript can read too.
- Check each page type. Test at least your homepage, a category page, a product page and an article. Apps and theme components differ by page type.
Quick check for each page
- The main text is in the page source, not only in Inspect.
- The title, meta description and headings are in the page source.
- Price, availability and specifications are in the page source.
- FAQ answers and tab texts are in the page source.
- The menu and internal links are ordinary links with an href.
- The page can be read with JavaScript turned off in the browser.
You will find more technical checks in the technical SEO checklist. Access for crawlers is managed separately, with robots.txt.
How Outseen approaches this
Atlas, the research engine developed by Outseen, reads the public HTML your website returns, including titles, headings, link text and supported JSON-LD. Anything that only appears after JavaScript falls outside that investigation. If information is missing from the HTML, that stays visible in the findings. When we build a new website, the SEO and GEO basics are part of the project.
What does this mean for your business?
Look at the page source of your most visited pages today and check whether the text customers read is there too. If it is, AI crawlers without JavaScript can read your page and you do not need to change anything. If something is missing, note which part it is and on which page types it happens, so your builder can fix it in a targeted way.
Bear in mind that public information about AI crawlers is limited and comes from a single measurement from late 2024. Content in the server HTML is the safe choice, even if crawlers start running JavaScript later: Google reads that content just as well, and visitors get a faster page.