How a search engine (and an AI) reads your page

A search engine and a generative AI don't "look" at your page the way a human visitor does. They fetch a stream of HTML, turn it (or not) into a data structure called the DOM, then extract raw text from it and split that text into chunks for indexing or for use as an answer. Understanding this path changes, concretely, how you should write and structure your pages.

The path from raw HTML to rendering

When a crawler visits a URL, it first receives an HTTP response containing the raw HTML sent by your server, before any JavaScript execution. Most bots analyze this initial HTML first, because fully executing it (the "render") costs computing time and memory, multiplied by the billions of pages crawled every day.

Google can render JavaScript in a second pass, through a separate queue that can take anywhere from a few seconds to several days depending on current load and the priority assigned to your site. In other words, there's a gap — sometimes significant — between when your raw HTML is read and when its fully rendered version (after JavaScript execution) is taken into account. Generative AI crawlers (ChatGPT, Claude, Perplexity in particular) are generally more limited on this point: they tend to rely more on static HTML or on versions already indexed by third-party search engines, rather than on a full, systematic JavaScript render on every visit.

This difference in handling explains why two tools can end up with a different picture of the same page: one may have waited for the full render, the other may not have.

What a crawler sees vs. what a browser sees

A browser executes CSS and JavaScript, loads fonts, applies animations, and displays a complete visual render, pixel by pixel. A crawler sees nothing visual: it processes a string of characters (the HTML) and, possibly, a tree structure built from that string.

Concretely, if content exists only visually — information encoded solely in an image with no alt text, text generated by a CSS animation, content injected after a user click, a tooltip that only appears on hover — it's likely that no bot will ever perceive it as usable text. It's not that it deliberately ignores it: it simply has no way to reproduce the human interaction that triggers its appearance.

The DOM and text extraction

The DOM (Document Object Model) is the tree representation of the HTML: each tag becomes a "node" with parents, children, and attributes. Engines and AIs extract usable text from this tree structure, relying in particular on:

• the tag hierarchy (<header>, <nav>, <main>, <article>, <footer>) to distinguish main content from peripheral elements;
• heading tags (<h1> through <h6>) to identify the logical structure of the content;
• the order of the HTML in the document (not the visual order imposed by CSS).

An often-overlooked point: if you use position: absolute or flex-direction: row-reverse to visually reorder blocks, the order read by the machine remains the one in the source code, not the one displayed on screen. A paragraph placed last in the HTML but visually moved to the top of the page via CSS will still be treated, by most extraction systems, as the last content block — which can distort the logical structure perceived by an automated reader.

JavaScript: the invisible-content trap

Sites built with client-side JavaScript (some React, Vue, or Angular apps without server-side rendering) carry a real risk: if the text content only appears after the script runs, and the bot visiting the page doesn't execute it or only partially does, that content is simply absent from what gets indexed. The risk is especially high for blocks loaded via aggressive lazy loading, content hidden behind a tab closed by default, or data injected through an asynchronous API call after the page's initial load.

Several approaches limit this risk:

<!-- Server-side rendering (SSR): the initial HTML already contains the text -->
<article>
  <h1>Title visible immediately in the source HTML</h1>
  <p>This paragraph is present before any JS runs.</p>
</article>

SSR (server-side rendering), static rendering (SSG), or progressive hydration guarantee that essential text is present from the very first HTTP response, without depending on the visiting bot executing JavaScript. These approaches aren't just technical options reserved for developers: they have a direct impact on how visible your content is to any automated system, whether a classic search engine or a generative AI browsing the web to build an answer.

What this concretely means for your content

Three direct implications for writing and building your pages:

1. Important text content (arguments, answers to questions, figures) must exist in the raw HTML, not only after interaction or animation.
2. The structure of the source code should reflect the logical reading order, independent of the visual formatting imposed by CSS.
3. Clear semantic markup helps separate main content from noise (menus, ads, repetitive footers) — a point covered in the lesson on semantic HTML5.

To check concretely what a bot can read, compare the raw HTML returned by your server (right-click → "view page source", not the inspector that shows the post-render DOM) with what's displayed on screen. Any notable gap — a block of text present visually but absent from the source code — is a signal to fix as a priority.

This question of the technical path content takes ties directly into two topics from the next module: how an LLM chunks your content once the text is extracted, and how to manage crawler access via the robots.txt file.