The website that doesn't show the same page to bots and humans
Same URL, two documents: a rich page for human visitors, a structured text version for AI chatbot crawlers. How content negotiation works, where to set it up, and the pitfalls to avoid so both versions stay consistent.
A visitor lands on a product page. Their browser loads a page built in JavaScript, the language that lets a site come alive, display an image carousel, show a price that changes with the selected color. Thirty seconds later, a ChatGPT crawler arrives at the same address. It sees none of that. No carousel, no price, sometimes an almost empty page, because this crawler does not stay long enough to let the JavaScript run and build the page in front of it. Two visitors, one address, two radically different experiences. This is where a large part of GEO plays out: optimizing a site so that it is visible and properly cited in the answers of AI chatbots such as ChatGPT, Claude, Gemini or Perplexity, rather than in Google's blue links.
The solution gaining ground right now is neither to block everything nor to throw the doors wide open. It is something subtler: showing a different face depending on who is looking. And this principle, which has a precise name in computing, deserves to be understood by anyone running a website today.
One address, two possible documents
The web runs on a protocol called HTTP, which governs exchanges between a browser and a server. This protocol has always included a mechanism called content negotiation. The idea is simple: a server can send different versions of the same document depending on what the requester asks for. An example everyone has come across without thinking about it: a site serves its page in French or in English depending on the browser's language, without changing the address shown in the bar at the top. The page keeps the same address for everyone, but its content adapts to whoever requests it.
Applied to GEO, the principle becomes: at a product page's address, a human's browser receives the rich, animated version designed for the eye. An AI chatbot's crawler receives a structured text version, designed to be read and understood in a single pass, without waiting for a script to run. Same URL, two documents. The human visitor never sees the text version. The crawler never sees the carousel.
An image helps make the idea stick. Picture a hotel that receives two kinds of visitors at the same front door: guests, and a safety inspector who has come to check the emergency exits. The staff do not give the inspector a tour of the presidential suite. They hand over the evacuation plan, the latest inspection report, the serial numbers of the fire extinguishers. Same building, same door, two journeys designed for two different needs. A site that uses content negotiation does exactly that: it identifies who is knocking at the door and adapts what it shows.
Why the alternative does not hold up
Before this practice existed, a site that wanted to be readable by AI chatbots had two options. The first: render everything server-side, that is, build the complete page, with its final content, before sending it to anyone, human or bot. This approach works, but it is costly in development time on modern sites built around JavaScript, and it slows down display for the human user waiting for the page.
The second option, more common by default: do nothing, and let the bot cope with a half-empty page. That is what happens on a huge number of e-commerce sites and product catalogs built with modern tools. The result shows in AI chatbot citations: brands missing from the answers even though their content really exists, simply because that content never reached the bot that could have read it.
Content negotiation resolves this dilemma without sacrificing either the human storefront or machine readability. The site stays as it is, with its animations and design, for human visitors. A layer is added that intercepts identified bots and serves them a structured text translation of the same information. Nobody loses anything.
How it gets set up in practice
Technically, this interception can happen in several places, and the choice depends on the site's infrastructure.
| Where it goes | What it does | Who it is for |
|---|---|---|
| An intermediary at the traffic provider (such as Cloudflare) | Intercepts the request before it reaches the server, detects the bot, returns the text | High-traffic sites that need speed |
| A small program added directly on the server (PHP, Apache, nginx) | Same logic, but handled within the site's existing infrastructure | Sites on traditional hosting, with no intermediary layer |
| A rule built into the framework (Next.js and equivalents) | The application's own code decides, at response time, whether to return the normal page or its text version | Sites built with a modern JavaScript framework |
In every case, detection relies on two clues the visitor sends without even thinking about it: the identity it declares (what is called the user-agent, a label every browser or bot attaches to its request to introduce itself) and sometimes what it says it is able to receive. GPTBot, PerplexityBot and ClaudeBot declare their identity openly, in line with the transparency their publishers advertise. The server reads that label and chooses which version to send.
What changes with this building block arriving in modern frameworks like Next.js is how easy it is to set up. Just a year ago, putting this logic in place often required custom development or going through an external intermediary. Today, it can be written as an ordinary routing rule in the application's own code, in the same place where you already handle redirecting a page that no longer exists or switching languages. The technical barrier collapses, and what used to be an infrastructure project becomes a line of configuration.
The trap to avoid: a text version that tells a different story
This mechanism is only worth anything if the version sent to the bot says the same thing as the version sent to the human. The main risk is not technical, it is editorial: letting the two versions drift apart over time. A team updates the price on the visible page, forgets to regenerate the text version, and the chatbot quotes a price that has not existed for three months. A chatbot that repeats false information because the source itself is contradictory is just doing its job. The fault lies with whoever let two versions of the same truth fall out of sync.
There is also a trickier trap: fake bots. Nothing stops someone from changing the identity their browser sends to pass as GPTBot and access the text version, which is faster to load and easier to copy. Detection based solely on the declared label is therefore never absolute security; it remains a convenience filter for legitimate bots, not a lock.
Finally, the text version must stay faithful to the content, not to its layout. There is a temptation to use this version to slip in sales arguments that are absent from the visible page, in the hope of influencing the chatbot's answer. That practice has a name: cloaking, which means showing bots different content from what visitors see. Google defines it as "the practice of presenting different content to users and search engines with the intent to manipulate search rankings and mislead users", and lists it among the practices covered by its spam policies: sites that violate them may rank lower or not appear in results at all (Google Search Central, spam policies). Content negotiation stays legitimate as long as both versions say the same thing. It turns into cloaking as soon as the version served to bots tells a different story. Google also states that its policies cover attempts to manipulate generative AI responses in Google Search. A brand that provides two different stories depending on the audience therefore risks far more than it stands to gain.
What it really shifts
Content negotiation does not make a brand more interesting than it is. It simply removes a purely technical obstacle between content that already exists and a bot that, without it, would never have read it. That makes it different in nature from most GEO levers: here you are not working on the quality of an argument or the credibility of a source, you are working on the plumbing that lets that argument reach the automated reader.
For a brand whose site relies heavily on JavaScript, e-commerce catalogs first and foremost, it is often the missing link between content work already done and a persistent absence from AI chatbot answers. The text is good. It simply never reached its destination.
Vurto sets up this content negotiation directly on the site's infrastructure, as a Cloudflare Worker, PHP module, Apache or nginx configuration, or Next.js middleware depending on the existing setup, so that every page keeps speaking its original language to humans and its structured language to AI chatbots, without ever letting the two fall out of sync.