The Next LLM Will Know Your Competitors. Will It Know You?

In 2023 and 2024, a wave of website owners — encouraged by coverage of AI companies scraping the web for training data — added blocks to their robots.txt files. The blocks targeted AI crawlers specifically: GPTBot (OpenAI’s training crawler), ClaudeBot (Anthropic’s), Google’s Extended crawler (used for Gemini training), and the Common Crawl bots that supply training data to most of the large models in production today.

The decision felt reasonable at the time. AI companies were consuming web content at scale to train commercial systems, and the websites whose content was being used had not consented and were not compensated. The blocks were a form of protest, a form of protection, or simply a precaution.

What many of those website owners did not consider is the second-order consequence: by blocking the AI crawlers, they removed themselves from the training datasets of every large language model trained after those blocks were added. The next version of every major AI model — the one currently being trained on 2025 and 2026 web data — will have no inherent knowledge of those businesses.

Their competitors, whose sites were not blocked, will be in the weights.

What it means to be “in the weights”

When a large language model is trained, it ingests billions of pages of text and encodes that information into the numerical parameters — the weights — that define what the model knows. This knowledge is not stored as a retrievable database. It is compressed into the model’s structure in a way that allows it to draw on that information when responding to queries, without explicitly referencing any source.

A business whose website was crawled and included in the training data is, in a meaningful sense, known by the model. Not completely, not with photographic accuracy — but the model has some encoded representation of that business: what it does, where it is, what kind of language it uses to describe its services. When a user asks the model a question that is relevant to that business, the model has something to draw on.

A business whose website was blocked from the training crawl is unknown. The model has no representation of it in its weights. When a relevant query arrives, the model has nothing to offer about that business except what it might infer from other sources — its name appearing in a review, a mention in a directory the crawler did reach, a social media post the crawler indexed. The website itself — the primary source of controlled, accurate information about the business — contributed nothing.

This is the “inherent knowledge” gap. It is not recoverable after the training run completes. The model is trained once (or periodically, on new data). The weights are fixed at the point the training data was collected. A business that was blocked during the relevant data collection window is not in that model’s knowledge base, regardless of what it does after the model is released.

The two-layer problem

The training data gap is the structural issue — the permanent absence from a model’s base knowledge. But there is a second layer that operates in real time, and the robots.txt block affects this one too.

AI-powered search tools — Google’s AI Overviews, Perplexity, ChatGPT Search, Microsoft Copilot — do not rely exclusively on model weights when they answer queries. They supplement the model’s base knowledge with live web retrieval: crawling the current web at the moment the query is asked, pulling relevant content, and synthesising an answer that combines what the model already knows with what it just fetched.

This is retrieval-augmented generation — the mechanism that allows AI search tools to answer questions about events that happened after the model’s training cutoff, and to give more current, specific answers than the base model alone could produce.

For a business, this means that even if a model has no base knowledge of them, an AI search tool could still surface them — if it can retrieve their website at query time. But if the robots.txt block that prevented the training crawl is still in place, the retrieval-time crawl is also blocked. The business is absent from base knowledge AND absent from real-time retrieval. Both layers are affected by the same file.

A business that blocked AI crawlers in 2024 may have achieved exactly the opposite of what it intended: not protection from AI systems, but invisibility within them.

The query that the model answers from memory

The clearest way to understand the competitive implication is to think about what happens when an AI model is asked a question without internet access — in a context where retrieval is not available, or where the model is working from its base knowledge alone.

“Can you recommend an interior designer in Umhlanga?” — The model responds with what it knows. If it has encoded knowledge of specific businesses in that category and location, it may name them. If it has no such knowledge, it gives a generic answer or recommends searching for one. The businesses it names, if any, are there because their websites were in the training data.

This scenario is not hypothetical. It is every voice assistant query, every AI chat response without search enabled, every AI-generated summary that draws on model weights rather than live retrieval. The businesses with inherent model knowledge are the ones with a chance of appearing. The ones without it do not exist in that response.

As AI interfaces become more prevalent in how people find and evaluate local businesses — and the trajectory of that shift is not in question, only its pace — the businesses whose content is baked into the models will have a structural advantage that compounds with every new model version trained on web data that includes them.

What structured AI accessibility looks like

Not blocking AI crawlers is the minimum. It is the threshold below which a business actively removes itself from AI knowledge. But minimum compliance is not competitive advantage.

The businesses that are most effectively represented in AI training data and real-time AI retrieval are the ones whose content is structured for machine comprehension, not just for human readability. The distinction matters because AI systems — both during training and during retrieval — extract structured information more reliably than they extract free-form prose.

A business whose website declares its name, address, services, service area, and contact information in structured schema markup — in a format that both Google’s systems and AI crawlers can parse unambiguously — is more completely represented in any training dataset that crawl reaches. Its entity signals are clean. Its category is explicit. Its geographic anchoring is precise.

A business whose same information is embedded in flowing paragraph text is less reliably extracted. The crawler reads it. The model may encode some of it. But the specificity and accuracy of the representation depends on how well the natural language processing in the training pipeline handles that particular prose — which varies.

The llms.txt file — a standard now implemented on every TVS-built site — takes this further. It is a plain-text document at the site root, structured for AI consumption specifically, that declares in unambiguous terms what the business is, what it does, where it operates, and how to engage it. It is not the robots.txt equivalent for AI in the blocking sense — it is the site map equivalent, a structured guide that tells AI crawlers exactly what the site contains and how to interpret it.

A business with a well-written llms.txt is giving the training crawler the most accessible, accurate, machine-readable description of itself that exists anywhere on the web. That representation goes into the training data as authored — not as an AI inference from ambiguous prose.

The window that is closing

The practical implication of this is time-bound in a specific way. The major AI models in production are trained on data collected up to their knowledge cutoff — typically months before the model is released. The models currently being trained on 2025 and 2026 web data will be released over the next twelve to twenty-four months.

The businesses whose sites are accessible to AI crawlers now, and whose content is structured for AI comprehension now, are building their position in those models’ weights as those training runs proceed. The businesses that are blocked, or whose content is poorly structured, are being excluded from those weights during the window when the data is being collected.

After the model is released, the weights are fixed. The next opportunity to be included is the next training run — months or years later.

The SEO parallel is useful here. Domain authority in traditional search takes years to build and is difficult to displace once established. AI model knowledge, encoded during training, has a similar structural persistence — a business embedded in the weights of a widely-deployed model has a knowledge-base presence that its competitors who were excluded cannot quickly recover.

The window is not closed. The training runs that will produce the next generation of widely-deployed models are proceeding now. The question for any business with an online presence is whether they are currently accessible to the crawlers running those training passes — or whether a robots.txt entry from two years ago is still silently removing them from the internet’s most consequential new knowledge systems.

---

TVS implements AI accessibility standards — including llms.txt and schema-structured business data — on every website it builds. If you want to know whether your current site is accessible to AI systems or blocked from them, run a visibility check or speak to us.